A method for improving the quality of character image generation based on model debiasing training

Abstract

The present invention provides a method for improving the quality of character image generation based on model debiasing training, comprising the following steps: S1. Extract text embedding: deploy a pre-trained entity relationship network to extract human-centered words from a text prompt for human image generation; S2. Freeze U-Net parameters and insert HcD layer; S3. Calculate combined attention map; S4. Dynamically adjust weights: According to the characteristics of different stages of U-Net, dynamically adjust loss weights to balance the impact of different stages on generated images; and S5. Update HcD layer parameters: According to the calculated overall optimization target, use an optimizer to update the parameters of the HcD layer to continuously optimize the model's ability to generate high-quality human images and output high-quality images. The method of the present invention can not only effectively solve the structural problems in portrait generation, improve the quality and accuracy of generated images, but also retain the expressive power and generation diversity of the original model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular, to a method for improving the quality of character image generation based on model debiasing training. Background Art

[0002] In recent years, the generation of text to image, or T2I, has become a rapidly developing and widely used field, and has produced many excellent model architectures and learning paradigms. In this field, the booming model architecture based on generative adversarial networks has provided a comparable benchmark for the quality of image generation and the diversity of generated objects in the field of image generation. The latest progress in diffusion models has significantly enhanced the ability to generate text to image. The efficient image generation capability of the model architecture based on the diffusion model comes from the structural denoising process of the system, which is the reasoning process of image generation. In this process, random noise is gradually refined into a logical image that conforms to the text description. For example, the StableDiffusion model, based on the latent space method, first converts the text description into an implicit representation, and then embeds it into the denoising process to gradually refine it into a specific image.

[0003] Among them, in the field of text-to-image generation, the generation of human images has always been the focus of most attention and interest. Because human images are an indispensable part in many fields such as film, television, fashion, entertainment and advertising, and are also one of the most prominent aspects of cost-saving effects in artificial intelligence applications, there are many studies in this area dedicated to improving the accuracy and rationality of the posture of human images. In order to generate more realistic and accurate human images in text descriptions in this field, a more effective direct way to solve such challenges is to use user restrictions in the reasoning stage, such as using ControlNet (Zhang L, Rao A, Agrawala M. Adding conditional controlto text-to-image diffusion models[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision. 2023: 3836-3847.) to use additional conditions of the human body structure to control the posture of the final generated human image to more accurately meet the text description. This method can significantly improve the structural integrity of human image generation, but this method relies on additional conditions and does not solve the inherent problems in the model. It also requires users to provide additional conditions in the reasoning stage, which is not user-friendly and difficult to use. Another feasible approach is to fine-tune the large model like using LoRA (Hu EJ, Shen Y, Wallis P, et al. Lora: Low-rank adaptation oflarge language models[J]. arXiv preprint arXiv:2106.09685, 2021.), and train it on a human-centric dataset so that the pre-trained model can more accurately express human characteristics. This method can enhance the human feature representation of the image, but it may modify the original expressive power of the model and cause catastrophic forgetting, causing the output to be limited by the characteristics of the fine-tuning dataset.

[0004] The difficulty of solving the above problems and defects is as follows: The generation of human images has always been the most important part in the field of text-to-image generation. Improving the accuracy and rationality of human image postures has always been an important research direction. The usual approach to this problem is to add additional conditions to restrict the reasoning stage of the model. We want to improve user ease of use. The difficulty lies in how to integrate human posture information into the reasoning process of the large model without allowing users to provide additional conditions in the reasoning stage. Our method can solve this problem well.

[0005] The significance of solving the above problems and defects is to improve the structural accuracy and detail information of the generated human images, so that the human images generated by the diffusion model are more consistent with the text description and more logical. Summary of the invention

[0006] The present invention provides a method for improving the quality of character image generation based on model debiasing training, which can not only effectively solve the structural problems in portrait generation and improve the quality and accuracy of generated images, but also retain the expressiveness and generation diversity of the original model.

[0007] The technical solution of the present invention is as follows: The method for improving the quality of character image generation based on model debiasing training of the present invention comprises the following steps: S1. extracting text embedding: deploying a pre-trained entity relationship network to extract human-centered words from a text prompt for human image generation; S2. freezing U-Net parameters and inserting HcD layer; S3. calculating the combined attention map; S4. dynamically adjusting weights: dynamically adjusting loss weights according to the characteristics of different stages of U-Net to balance the impact of different stages on the generated image; and S5. updating HcD layer parameters: according to the calculated overall optimization target, using an optimizer to update the parameters of the HcD layer to continuously optimize the model's ability to generate high-quality human images and output high-quality images.

[0008] Optionally, in the above-mentioned method for improving the quality of character image generation based on model debiasing training, in step S1, the input text prompt is processed by a word segmenter to obtain a text embedding, in preparation for the subsequent use of text information in a diffusion model; then the human-centered text embedding is further extracted by a human-centered text extractor, in preparation for the subsequent generation of a human-centered attention map in the diffusion model.

[0009] Optionally, in the above-mentioned method for improving the quality of character image generation based on model debiasing training, in step S2, the U-Net parameters are frozen in the latent space of the pre-trained stable diffusion model so that they are not updated with training to maintain the original expressive ability of the model, and then a human-centric debiasing (HcD) layer is inserted into the U-Net framework with frozen parameters.

[0010] Optionally, in the above-mentioned method for improving the quality of character image generation based on model debias training, in step S3, the human-centered word Token text extracted in step S1 is embedded, and a human-centered attention map is generated through the calculation attention of the HcD layer. Combined with the original attention map, the forward attention map of the cross-attention layer in the pre-trained denoising network is defined as a combination of the human-centered attention map and the original attention map.

[0011] Optionally, in the above-mentioned method for improving the quality of character image generation based on model debiasing training, in step S3, in the cross-attention mechanism, the text embedding is processed by a human-centered debiasing (HcD) layer to obtain an additional “Key” , With implicit representation Interaction-based human-centered attention map generation , and then according to the formula Compute the combined attention map ,in is the attention combination weight.

[0012] Optionally, in the above-mentioned method for improving the quality of character image generation based on model debiasing training, in step S4, according to the resolution scale and time step characteristics of different stages of U-Net, the loss weight λ is dynamically adjusted using a cosine function to balance the impact of different stages on the generated image.

[0013] According to the technical solution of the present invention, the beneficial effects produced are: The present invention realizes an image generation method based on model debiasing training to improve the generation ability of diffusion models for human images. In view of the problems of inaccurate structure and uncoordinated body proportions when generating portraits by existing text-to-image diffusion models, as well as the limitations brought by additional conditions or fine-tuning methods, the present invention directly integrates human-centered debiasing information in the model fine-tuning stage by introducing a human-centered debiasing (HcD) layer, without adding additional conditions in the reasoning stage. The method of the present invention can not only effectively solve the structural problems in portrait generation, improve the quality and accuracy of generated images, but also retain the expressive power and generation diversity of the original model. In the prior art, although the use of additional conditions can improve the structure of portraits, it increases the complexity of reasoning and the dependence on additional data; and the fine-tuning method may change the original expressive power of the model and cause forgetting problems. Through in-depth analysis of the cross-attention layer, the method of the invention designs a human center alignment loss, aligns the human center de-biased information such as the key posture graph with the HcD layer, so that the model can focus more on the generation of human structure while avoiding adverse effects on the original capabilities of the model. It can significantly improve the effect of portrait generation without changing the overall architecture and expression capabilities of the model, and better meet users' needs for high-quality and personalized portrait generation.

[0014] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0016] Figure 1 It is a flow chart of a method for improving the quality of character image generation based on model debiasing training of the present invention; Figure 2 It is a schematic diagram of debiasing training of the human-centered embedded diffusion model of the present invention; Figure 3 is a schematic diagram of the human body center alignment loss process of the present invention; Figure 4 It is a comparison diagram of human images obtained by the original diffusion model and the method of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical method and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. These examples are only illustrative and not limiting of the present invention.

[0018] The method for improving the quality of character image generation based on model debiasing training of the present invention is based on a diffusion model and effectively utilizes the human body center debiasing method to integrate it into the training stage of the text generation image model, thereby achieving more accurate posture control of character image generation and does not require the user to add additional control in the reasoning stage.

[0019] The present invention uses a human-centered debiasing layer, called the HcD layer, in the model training stage. The method of the present invention is a plug-and-play method for improving the quality of human image generation in human-centered debiasing training for diffusion models. Due to the important role of the cross-attention map in the diffusion model, the human-centered debiasing (HcD) layer is inserted into the cross-attention layer of the diffusion model. The implementation principle is to enhance the model's sensitivity to human-centered text information during the denoising process of the diffusion model, thereby improving the structural accuracy and detail information of the generated human image. The human-centered debiasing (HcD) layer is an additional learning module to enhance the interaction between implicit features and human-centered text information in the cross-attention map. The module is integrated without changing the pre-existing expressive power of the cross-attention layer, and its parameters are kept frozen during training. The present invention enhances human image generation in the diffusion model without additional conditions in the inference stage, thereby ensuring high-quality generation of human images.

[0020] The principle of the present invention is: utilizing the characteristics of a human-centered debiasing (HcD) layer to optimize parameters through a human-centered alignment loss, inserting the HcD layer into a frozen U-Net framework in a large model, and then integrating it into the denoising process of the entire diffusion model, thereby refining the human-centered structural graph information into the image generation process, thereby improving the diffusion model's ability to represent the human structure in the training stage, while retaining the diffusion model's own ability to generate diversity and richness of images, without providing additional conditions in the reasoning stage.

[0021] The method of the present invention is a method for improving the quality of character image generation by body-centered debiasing training. By inserting a body-centered debiasing (HcD) layer into a U-Net framework of a frozen diffusion model, body-centered words in text prompts are extracted, and only the indicators corresponding to these words are trained in an attention map. In this selection method, the training of the attention map focuses on relevant areas centered on the body, and then a pre-trained encoder is used to extract features from a human key posture map, and these features are aligned with the body-centered attention map, thereby refining the relevant areas centered on the body, and removing the body structure bias information of the generated image with the body as the center, so that the diffusion model significantly improves the generation ability of human images and generates high-quality human images.

[0022] like Figure 1 As shown, the method for improving the quality of character image generation based on model debiasing training of the present invention comprises the following steps: S1. Extracting Text Embeddings: Deploy a pre-trained entity relation network to extract human-centric terms from a text prompt for human image generation.

[0023] Specifically, the input text prompt is processed through a Tokenizer to obtain text embedding, in preparation for the subsequent use of text information in the diffusion model; then the human-centered text embedding is further extracted through a human-centered text extractor, in preparation for the subsequent generation of a human-centered attention map in the diffusion model.

[0024] S2. Freeze U-Net parameters and insert HcD layer: Freeze the U-Net parameters in the latent space of the pre-trained stable diffusion model so that they are not updated with training to maintain the original expressiveness of the model. Then, a learnable HcD layer (human-centered debiasing (HcD) layer) is inserted into the U-Net framework with frozen parameters. The HcD layer consists of multiple MLP networks to enhance the model's sensitivity to human-centered text information.

[0025] Figure 2The human-centered debiasing (HcD) layer is inserted into the frozen pre-trained diffusion model’s U-Net framework. The figure shows the denoising process of the diffusion model. Random noise generates an image through the step-by-step denoising process of U-Net. An additional learnable module, the human-centered debiasing (HcD) layer, is inserted into the cross-attention module in the U-Net network. The human-centered word Token text extracted in the previous step is embedded, and a human-centered attention map is generated after the calculation attention of the HcD layer. Combined with the original attention map, a combined attention map of the human-centered attention map and the original attention map is obtained. The forward attention map of the cross-attention layer in the pre-trained denoising network is defined as this combined attention map, thereby integrating the human-centered features into the denoising process of the diffusion model.

[0026] S3. Calculate the combined attention map: embed the human-centered word token text extracted in step S1, generate a human-centered attention map through the calculation attention of the HcD layer, and combine it with the original attention map. Define the forward attention map of the cross-attention layer in the pre-trained denoising network as the combination of the human-centered attention map and the original attention map.

[0027] Specifically, in the cross-attention mechanism, the text embedding is processed through the HcD layer to obtain an additional "Key" , With implicit representation Interaction-based human-centered attention map generation , and then according to the formula Compute the combined attention map ,in The combined attention map is used to guide image generation to focus more on human structure-related information.

[0028] S4. Dynamically adjust weights: According to the characteristics of different stages of U-Net, dynamically adjust the loss weights to balance the impact of different stages on the generated image.

[0029] Specifically, according to the resolution scale and time step characteristics of different stages of U-Net (such as down, mid, and up), the cosine function is used to dynamically adjust the loss weight λ to balance the impact of different stages on the generated image.

[0030] S5. Update HcD layer parameters: Based on the calculated overall optimization goal, use the optimizer to update the parameters of the HcD layer to continuously optimize the model's ability to generate high-quality human images and output high-quality images.

[0031] Specifically, according to the calculated overall optimization goal (including body center alignment loss and denoising loss ), and use an optimizer (such as AdamW) to update the parameters of the HcD layer to improve the model's ability to generate high-quality portrait images.

[0032] like Figure 3 The process of realizing the human center alignment loss is based on the human-centered debiasing (HcD) layer. Figure 2 The human-centered attention map generated by the extracted text embedding after the computational attention of the HcD layer is aligned with the feature map generated by extracting features from the human key posture map, thereby making the features more closely fit the human-centered text description during the diffusion process.

[0033] The present invention provides an innovative method for improving the quality of human image generation. By effectively utilizing the debiasing information centered on the human body in the diffusion model, it is committed to solving the shortcomings of existing methods in portrait generation. The method first builds a high-quality and diverse dataset with human characteristics on a dataset containing rich human images and performs careful preprocessing. Then, based on the key role of the cross-attention layer in the diffusion model in portrait generation, targeted improvement measures are proposed. By introducing the HcD layer, the sensitivity of the model to human-related information is enhanced, and a unique human-centered alignment loss is designed to ensure that the model can accurately capture and utilize the human-centered debiasing information. This method not only significantly improves the quality and accuracy of generated portraits, but also retains the original expression ability and generation diversity of the model, bringing a more efficient and high-quality solution to the field of portrait generation. The specific implementation steps are as follows:

[0034] S1. Extract text embedding: For a text prompt: "A lady in a red dress is dancing". First, use a specific Tokenizer to process this text prompt. The Tokenizer breaks the text into individual words or subwords and assigns a unique identifier to each word or subword. Then, these identifiers are converted into vector representations through a pre-trained word vector model or other encoding methods to obtain the text embedding of the entire text prompt. Next, the obtained text embedding is input into the human-centric text extractor. This extractor identifies the parts of the text related to the human body, such as "lady", "dancing", etc., based on predefined rules or models. Then, these parts related to the human body are further processed to extract the human-centric text embedding.

[0035] S2. Freeze U-Net parameters and insert HcD layer: In the pre-trained latent diffusion model, find the parameters of the U-Net part and set them to a non-updatable state. This means that in the subsequent training process, the parameters of the U-Net will remain unchanged, thereby retaining the original expressive power of the model. Insert the HcD layer at the appropriate position in the U-Net architecture. This layer consists of multiple MLP networks, and the specific structure and parameter settings are based on the definition in the paper. Calculate the relevant quantity: Let the potential representation of the input be , through the weight matrix Convert it to a query matrix ,Right now The human-centered text embedding is denoted as , processed by the HcD layer to obtain an additional "Key" ,Right now ,in Represents the transformation operation of the HcD layer.

[0036] S3. Calculate the combined attention map: According to the formula Computing Human Central Attention Map ,in is the potential projection dimension. Let the original attention map be , according to the formula Compute the combined attention map ,in is the attention combination weight. For the potential representation that has been obtained and text embedding , calculated through the above steps , , and ,These quantities will play a key role in the subsequent image generation process, guiding the model to pay more attention to information related to the human body, thereby improving the quality of generated portraits.

[0037] S4. Dynamically adjust weights: Divide the U-Net stage and clarify the three stages of U-Net: "down" (downsampling, resolution reduction, more feature maps, extracting the overall contour, etc.), "mid" (bridging the gap, condensing features to focus on key human structures), and "up" (upsampling, increasing resolution, reconstructing details). Determine the time step and cosine function adjustment rules: Set the total time step . In the “down” phase, the weight , start hour, , which is conducive to capturing broad features, hour, ; "mid" stage, The starting value is 1. Time ; "up" stage, , Time , Time , adapt to the focus of each stage. Weight application and feedback adjustment, training iteration, according to the current Calculate the weights of each stage for the loss function Calculate. Evaluate the generated image every 100 iterations and fine-tune the cosine function parameters (like values, etc.) to produce high-quality images that fit the text.

[0038] S5. Update HcD layer parameters: Clarify the overall optimization goal , human center alignment loss Quantify the difference between the generated image and the text human features and key pose references, denoising loss Measure the difference from the noise-free target image, overall ( is the pre-adjusted weight). Select AdamW optimizer, and each training iteration is based on its algorithm. Calculate the gradient of the HcD layer parameters, combine the learning rate (such as 0.001) and momentum parameters, and update the parameters according to the rules to improve the model's ability to fit human-centric text. Perform each step in multiple rounds of iterations. After the end, use SSIM and perceptual loss evaluation, as well as manual subjective evaluation, to monitor the improvement of the generated image quality. Continue until the convergence conditions are met (such as The value is lower than 0.01 or the indicator improves slightly over multiple rounds) to train the model to output high-quality, description-fitting human images based on text prompts through the diffusion model denoising process.

[0039] The proposed method uses a human-centered debiasing (HcD) layer to integrate human-centered debiasing information (such as key poses) during the diffusion model fine-tuning stage through the human-centered debiasing loss alignment method. This effectively solves many limitations of existing methods, such as using additional conditions to increase reasoning complexity, limiting the creation of diverse images, and fine-tuning methods changing the original expressiveness of the model. This method can significantly improve the quality and accuracy of human image generation without adding additional conditions in the reasoning stage, and can also preserve the expressiveness and generation diversity of the original diffusion model.

[0040] Figure 4This is a comparison chart of the experimental results of human images obtained by the original diffusion model and the method of the present invention. The left side is a human image with unreasonable human body structure generated by the original diffusion model, and the right side is a human image with improved quality obtained after the HcD layer debiasing training of the present invention.

[0041] The present invention provides a high-quality human image generation method that improves the performance of the existing diffusion model for the accuracy of human body structure and the rationality of posture. By freezing the parameters of the U-Net in the denoising process of the diffusion model and inserting a human-centered debiasing (HcD) layer, the information of the key posture diagram centered on the human body that can effectively represent the human body structure is integrated into the denoising process of the diffusion model, thereby enhancing the human image generation capability of the diffusion model, and further improving the human image generated by the diffusion model to more accurately reflect the text information and improve the rationality of the human body posture. This method no longer uses the additional information provided by the user in the inference stage to improve the accuracy of the generated image through the previous method such as ControlNet, but removes the bias information based on the model training process to achieve a more accurate model with better performance in human posture and details, so that the image generated by the diffusion model can meet higher standards in applications in the fields of film and television, fashion, entertainment and advertising.

[0042] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the protection scope of the present claims, and other implementations and combinations of implementations according to the concept of the present invention belong to the protection scope of the present invention.

Claims

1. A method for improving the quality of character image generation based on model debiasing training, characterized in that: The following steps are involved: S1. Extracting text embeddings: Deploy a pre-trained entity relation network to extract human-centric terms from a text prompt for human image generation. S2. Freeze U-Net parameters and insert a human-centric debiasing (HcD) layer; S3. Calculate the combined attention map; S4. Dynamically adjust weights: According to the characteristics of different stages of U-Net, dynamically adjust the loss weights to balance the impact of different stages on the generated image; as well as S5. Update the parameters of the human-centered debiasing (HcD) layer: According to the calculated overall optimization goal, use the optimizer to update the parameters of the human-centered debiasing (HcD) layer to continuously optimize the model's ability to generate high-quality human images and output high-quality images.

2. The method for improving the quality of character image generation based on model debiasing training according to claim 1, characterized in that: In step S1, the input text prompt is processed by a word segmenter to obtain text embedding, in preparation for the subsequent use of text information in the diffusion model; then the human-centered text embedding is further extracted by a human-centered text extractor, in preparation for the subsequent generation of a human-centered attention map in the diffusion model.

3. The method for improving the quality of character image generation based on model debiasing training according to claim 1, characterized in that: In step S2, the U-Net parameters are frozen in the latent space of the pre-trained stable diffusion model so that they are not updated with training to maintain the original expressiveness of the model, and then the human-centric debiasing (HcD) layer is inserted into the U-Net framework with frozen parameters.

4. The method for improving the quality of character image generation based on model debiasing training according to claim 1, characterized in that: In step S3, the human-centered word token text extracted in step S1 is embedded, and a human-centered attention map is generated through the computational attention of the human-centered debiasing (HcD) layer. Combined with the original attention map, the forward attention map of the cross-attention layer in the pre-trained denoising network is defined as a combination of the human-centered attention map and the original attention map.

5. The method for improving the quality of character image generation based on model debiasing training according to claim 1, characterized in that: In step S3, in the cross-attention mechanism, the text embedding is processed through the human-centric debiasing (HcD) layer to obtain an additional "Key" , With implicit representation Interaction-based human-centered attention map generation , and then according to the formula Compute the combined attention map ,in is the attention combination weight.

6. The method for improving the quality of character image generation based on model debiasing training according to claim 1, characterized in that: In step S4, according to the resolution scale and time step characteristics of different stages of U-Net, the loss weight λ is dynamically adjusted using the cosine function to balance the impact of different stages on the generated image.

Images