Model training method and device, image generation method and device and electronic equipment

Abstract

The invention discloses a model training method and device, electronic equipment and a readable storage medium, and belongs to the technical field of artificial intelligence and electronic equipment. The method comprises the steps that a sample image set is determined based on an initial sample image set, and the sample image set comprises at least one sample image; wherein each sample image comprises a key area, the key area comprises a human face area and a shoulder area, the width and height of the human face area are both greater than or equal to a first threshold value, the human face area has pore features, and the human face area does not have defect features; and training the initial model based on the sample image set to obtain an image generation model.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a model training method, an image generation method, a device, and an electronic device. Background Technology

[0002] The rapid development of artificial intelligence (AI) technology has spurred the significant demand for AI-powered image creation. Currently, image generation models, such as denoising diffusion probabilistic models (DDPM), are widely used in the field of user profile creation due to their stability and efficiency.

[0003] In related technologies, when training image generation models, due to limitations in input size and computational resources, low-resolution images are typically used for training. This results in insufficient skin texture details learned by the image generation model, leading to poor skin texture in the generated images, thus impairing the training effect and consequently, the quality of the generated images. Summary of the Invention

[0004] The purpose of this application is to provide a model training method, apparatus, and electronic device that can improve the realism of skin texture while effectively avoiding the generation of defects, thereby improving the quality of the model-generated images.

[0005] In a first aspect, embodiments of this application provide a model training method, which includes: determining a sample image set based on an initial sample image set, the sample image set including at least one sample image; wherein each sample image includes a key region, the key region including a face region and a shoulder region, the width and height of the face region being greater than or equal to a first threshold, the face region having pore features and no blemish features; and training an initial model based on the sample image set to obtain an image generation model.

[0006] In some embodiments of this application, before determining the sample image set based on the initial sample image set, the following steps are further included:

[0007] Obtain an initial sample image set, which includes at least one candidate sample image;

[0008] Based on the initial sample image set, the sample image set is determined, including:

[0009] Keypoint detection is performed on each candidate sample image to identify key regions in each candidate sample image;

[0010] Based on the width and height of the key region in each candidate sample image, the key region is expanded outward, and the expanded key region is determined as the cropping region.

[0011] The cropped region of each candidate sample image is extracted to obtain the cropped sample image set;

[0012] The cropped sample image set is input into the multimodal scoring model, and the image quality information of each candidate sample image is output. The image quality information indicates whether there are pore features and blemish features in the face region of each candidate sample image.

[0013] The sample image set is determined based on the image quality information of each candidate sample image.

[0014] In some embodiments of this application, a sample image set is determined based on the image quality information of each candidate sample image, including:

[0015] The sample image set is determined by all candidate sample images that meet the first condition;

[0016] The first condition includes: the image quality information indicates that the face region of the candidate sample image has pore features and no blemish features, and the width and height of the face region of the candidate sample image are both greater than or equal to the first threshold.

[0017] In some embodiments of this application, before training the initial model based on the sample image set, the method further includes:

[0018] Each sample image in the sample image set is scaled and cropped to generate a first-size image, a second-size image, and a third-size image corresponding to each sample image. The image size of the first-size image is smaller than that of the second-size image, and the image size of the second-size image is smaller than that of the third-size image.

[0019] All images of the first size are identified as the first sample image set, all images of the second size are identified as the second sample image set, and all images of the third size are identified as the third sample image set;

[0020] The initial model is trained based on the sample image set, including:

[0021] The initial model is trained based on the first, second, and third sample image sets.

[0022] In some embodiments of this application, the initial model is trained based on a sample image set, including:

[0023] All second-size images in the second sample image set are input into the initial model. The Unet module of the initial model is used to perform feature mapping processing on each second-size image to obtain the first predicted feature information of each second-size image.

[0024] The model parameters of the Unet module are updated based on the first difference information between the first predicted feature information of each second-size image and the reference feature information of each second-size image.

[0025] The reference feature information for each second-size image is obtained by performing latent space feature mapping on each second-size image through the VAE encoder of the image generation module.

[0026] In some embodiments of this application, after updating the model parameters of the Unet module based on the first difference information between the first predicted feature information of each second-size image and the reference feature information of each second-size image, the method further includes:

[0027] Input at least one third-size image from the third sample image set into the image generation model, and perform feature mapping processing on each third-size image through the Unet module to obtain the second predicted feature information of each third-size image;

[0028] The model parameters of the Unet module are updated based on the second difference information between the second predicted feature information and the reference feature information of the third-size image.

[0029] The reference feature information for each third-size image is obtained by performing latent space feature mapping on each third-size image through the VAE encoder of the image generation module.

[0030] In some embodiments of this application, after updating the model parameters of the Unet module based on the second difference information, the method further includes:

[0031] Based on the updated model parameters of the Unet module, a reference Unet module is constructed;

[0032] Input the reference feature information of the third-size image corresponding to each sample image into the reference Unet module, and output the third predicted feature information;

[0033] The reference feature information of the first-size image and the reference feature information of the third-size image corresponding to each sample image are fused to output the fused feature information.

[0034] The fused feature information is input into the Unet module, which outputs the fourth predicted feature information.

[0035] The weight parameters of the Unet module are updated based on the difference between the third and fourth predicted feature information.

[0036] In some embodiments of this application, after updating the weight parameters of the Unet module based on the difference information between the third and fourth predicted feature information, the method further includes:

[0037] Based on the first-size image and the second-size image corresponding to each sample image, at least one first image sample pair is constructed, and based on the second-size image and the third-size image corresponding to each sample image, at least one second image sample pair is constructed;

[0038] Input at least one first image sample pair into the initial model, and perform feature mapping processing on the at least one first image sample pair through the Unet module to output the fifth predicted feature information;

[0039] Input at least one second image sample pair into the initial model, and perform feature mapping processing on the at least one second image sample pair through the reference Unet module to output the sixth predicted feature information;

[0040] The model parameters of the Unet module are updated based on the difference between the fifth and sixth predicted feature information.

[0041] In some embodiments of this application, before training the initial model based on the sample image set, the method further includes:

[0042] Obtain at least one candidate image from the sample image set whose face region size is greater than the second size and whose face region proportion is greater than the second threshold;

[0043] Obtain the complete face image and partial face image corresponding to each candidate image;

[0044] The initial model is trained based on the sample image set, including:

[0045] The initial model is trained based on the complete face image and partial face image corresponding to each candidate image.

[0046] In some embodiments of this application, obtaining the complete face image and partial face image corresponding to each candidate image includes:

[0047] Each candidate image is input into the face detection model, and the facial feature point information of each candidate image is output.

[0048] Based on the facial feature point information of each candidate image, the face is cropped from each candidate image to obtain the complete face image corresponding to each candidate image.

[0049] The complete face image corresponding to each candidate image is input into the fine segmentation model, and the binary mask image corresponding to each complete face image is output. The image pixel values ​​of the binary mask image include white area pixel values ​​and black area pixel values. The image region corresponding to the white area pixel values ​​is the face skin region, and the image region corresponding to the black area pixel values ​​is the non-face skin region.

[0050] The image regions in all complete face images that correspond to the mask image regions that satisfy the second condition are determined as the sampling regions; wherein, the second condition is that the image pixel values ​​of the mask image regions in the binary mask image are the white area pixel values;

[0051] According to the reference cropping size, the sampled area of ​​each complete face image is cropped, and the cropped image is determined as the local face image corresponding to each candidate image.

[0052] In some embodiments of this application, an initial model is trained based on the complete face image and partial face image corresponding to each candidate image, including:

[0053] Based on the complete face image and partial face image corresponding to each candidate image, construct at least one set of face images;

[0054] Input at least one set of face images into the initial model and output predicted face feature information;

[0055] The model parameters of the initial model are updated based on the differences between the predicted facial feature information and the feature information of all complete facial images.

[0056] In some embodiments of this application, initial training is performed based on a sample image set, including:

[0057] Each candidate image is resized to a first size to obtain a first quality image. The first quality image is then blurred, and the image size of the blurred first quality image is subsequently resized to a second size and then back to the first size to obtain a second quality image. The first size is larger than the second size.

[0058] Establish a correlation between a first quality image and a first image detail cue, and establish a correlation between a second quality image and a second image detail cue, and construct at least one image sample pair based on the first quality image and the second quality image, wherein the first image detail cue and the second image detail cue indicate image detail features;

[0059] At least one image sample pair is input into the initial model. The Unet module performs feature mapping processing on the at least one image sample pair to obtain the seventh predicted feature information of the first quality image and the eighth predicted feature information of the second quality image.

[0060] The model parameters of the initial model are updated based on the difference between the seventh and eighth predicted feature information.

[0061] Secondly, embodiments of this application provide an image generation method, which includes: receiving an original image and prompt words input by a user; associating the original image and prompt words with an input image generation model to perform model inference; and outputting a model-generated image.

[0062] In some embodiments of this application, the image generation model includes a VAE encoder, a text encoder, a Unet module, and a VAE decoder;

[0063] The original image and the prompt word are associated and input into the image generation model of any one of claims 1-12 for model inference, and the model-generated image is output, including:

[0064] The original image and the prompt words are associated with the input image generation model. The original image is then encoded into image feature information through a VAE encoder.

[0065] The prompt words are encoded into text feature information using a text encoder;

[0066] Image and text feature information are correlated and input into the Unet module for feature mapping processing, and predicted feature information is output.

[0067] The VAE decoder decodes the predicted feature information and outputs the model-generated image.

[0068] Thirdly, embodiments of this application provide a model training apparatus, which includes: a processing module, configured to: determine a sample image set based on an initial sample image set, the sample image set including at least one sample image; wherein each sample image includes a key region, the key region including a face region and a shoulder region, the width and height of the face region being greater than or equal to a first threshold, the face region having pore features and no blemish features; and train an initial model based on the sample image set to obtain an image generation model.

[0069] In some embodiments of this application, the processing module is further configured to obtain an initial sample image set, which includes at least one candidate sample image, before determining the sample image set based on the initial sample image set;

[0070] The processing module is specifically used for:

[0071] Keypoint detection is performed on each candidate sample image to identify key regions in each candidate sample image;

[0072] Based on the width and height of the key region in each candidate sample image, the key region is expanded outward, and the expanded key region is determined as the cropping region.

[0073] The cropped region of each candidate sample image is extracted to obtain the cropped sample image set;

[0074] The cropped sample image set is input into the multimodal scoring model, and the image quality information of each candidate sample image is output. The image quality information indicates whether there are pore features and blemish features in the face region of each candidate sample image.

[0075] The sample image set is determined based on the image quality information of each candidate sample image.

[0076] In some embodiments of this application, the processing module is specifically used to determine a sample image set from all candidate sample images that satisfy the first condition;

[0077] The first condition includes: the image quality information indicates that the face region of the candidate sample image has pore features and no blemish features, and the width and height of the face region of the candidate sample image are both greater than or equal to the first threshold.

[0078] In some embodiments of this application, the processing module is further configured to scale and crop each sample image in the sample image set before training the initial model based on the sample image set, to generate a first-size image, a second-size image, and a third-size image corresponding to each sample image, wherein the image size of the first-size image is smaller than the image size of the second-size image, and the image size of the second-size image is smaller than the image size of the third-size image.

[0079] The processing module is also used to determine all first-size images as the first sample image set, all second-size images as the second sample image set, and all third-size images as the third sample image set;

[0080] The processing module is specifically used to train the initial model based on the first sample image set, the second sample image set, and the third sample image set.

[0081] In some embodiments of this application, the processing module is specifically used for:

[0082] All second-size images in the second sample image set are input into the initial model. The Unet module of the initial model is used to perform feature mapping processing on each second-size image to obtain the first predicted feature information of each second-size image.

[0083] The model parameters of the Unet module are updated based on the first difference information between the first predicted feature information of each second-size image and the reference feature information of each second-size image.

[0084] The reference feature information for each second-size image is obtained by performing latent space feature mapping on each second-size image through the VAE encoder of the image generation module.

[0085] In some embodiments of this application, the processing module is further configured to update the model parameters of the Unet module based on the first difference information between the first predicted feature information of each second-size image and the reference feature information of each second-size image, and then input at least one third-size image from the third sample image set into the image generation model, and perform feature mapping processing on each third-size image through the Unet module to obtain the second predicted feature information of each third-size image.

[0086] The processing module is also used to update the model parameters of the Unet module based on the second difference information between the second predicted feature information and the reference feature information of the third-size image;

[0087] The reference feature information for each third-size image is obtained by performing latent space feature mapping on each third-size image through the VAE encoder of the image generation module.

[0088] In some embodiments of this application, the processing module is further configured to:

[0089] After updating the model parameters of the Unet module based on the second difference information, a reference Unet module is constructed based on the updated model parameters of the Unet module.

[0090] Input the reference feature information of the third-size image corresponding to each sample image into the reference Unet module, and output the third predicted feature information;

[0091] The reference feature information of the first-size image and the reference feature information of the third-size image corresponding to each sample image are fused to output the fused feature information.

[0092] The fused feature information is input into the Unet module, which outputs the fourth predicted feature information.

[0093] The weight parameters of the Unet module are updated based on the difference between the third and fourth predicted feature information.

[0094] In some embodiments of this application, the processing module is further configured to:

[0095] After updating the weight parameters of the Unet module based on the difference information between the third and fourth predicted feature information, at least one first image sample pair is constructed based on the first and second size images corresponding to each sample image, and at least one second image sample pair is constructed based on the second and third size images corresponding to each sample image.

[0096] Input at least one first image sample pair into the initial model, and perform feature mapping processing on the at least one first image sample pair through the Unet module to output the fifth predicted feature information;

[0097] Input at least one second image sample pair into the initial model, and perform feature mapping processing on the at least one second image sample pair through the reference Unet module to output the sixth predicted feature information;

[0098] The model parameters of the Unet module are updated based on the difference between the fifth and sixth predicted feature information.

[0099] In some embodiments of this application, the processing module is further configured to:

[0100] Before training the initial model based on the sample image set, at least one candidate image is obtained from the sample image set whose face region size is greater than the second size and whose face region proportion is greater than the second threshold.

[0101] Obtain the complete face image and partial face image corresponding to each candidate image;

[0102] The initial model is trained based on the sample image set, including:

[0103] The initial model is trained based on the complete face image and partial face image corresponding to each candidate image.

[0104] In some embodiments of this application, the processing module is specifically used for:

[0105] Each candidate image is input into the face detection model, and the facial feature point information of each candidate image is output.

[0106] Based on the facial feature point information of each candidate image, the face is cropped from each candidate image to obtain the complete face image corresponding to each candidate image.

[0107] The complete face image corresponding to each candidate image is input into the fine segmentation model, and the binary mask image corresponding to each complete face image is output. The image pixel values ​​of the binary mask image include white area pixel values ​​and black area pixel values. The image region corresponding to the white area pixel values ​​is the face skin region, and the image region corresponding to the black area pixel values ​​is the non-face skin region.

[0108] The image regions in all complete face images that correspond to the mask image regions that satisfy the second condition are determined as the sampling regions; wherein, the second condition is that the image pixel values ​​of the mask image regions in the binary mask image are the white area pixel values;

[0109] According to the reference cropping size, the sampled area of ​​each complete face image is cropped, and the cropped image is determined as the local face image corresponding to each candidate image.

[0110] In some embodiments of this application, the processing module is specifically used for:

[0111] Based on the complete face image and partial face image corresponding to each candidate image, construct at least one set of face images;

[0112] Input at least one set of face images into the initial model and output predicted face feature information;

[0113] The model parameters of the initial model are updated based on the differences between the predicted facial feature information and the feature information of all complete facial images.

[0114] In some embodiments of this application, the processing module is specifically used for:

[0115] Each candidate image is resized to a first size to obtain a first quality image. The first quality image is then blurred, and the image size of the blurred first quality image is subsequently resized to a second size and then back to the first size to obtain a second quality image. The first size is larger than the second size.

[0116] Establish a correlation between a first quality image and a first image detail cue, and establish a correlation between a second quality image and a second image detail cue, and construct at least one image sample pair based on the first quality image and the second quality image, wherein the first image detail cue and the second image detail cue indicate image detail features;

[0117] At least one image sample pair is input into the initial model. The Unet module performs feature mapping processing on the at least one image sample pair to obtain the seventh predicted feature information of the first quality image and the eighth predicted feature information of the second quality image.

[0118] The model parameters of the initial model are updated based on the difference between the seventh and eighth predicted feature information.

[0119] Thirdly, embodiments of this application provide an image generation apparatus, which includes: an acquisition module and a processing module; the acquisition module is used to receive an original image and prompt words input by a user; the processing module is used to associate the original image and prompt words received by the acquisition module with an input image generation model to perform model inference and output a model-generated image.

[0120] In some embodiments of this application, the image generation model includes a VAE encoder, a text encoder, a Unet module, and a VAE decoder; the processing module is specifically used for:

[0121] The original image and the prompt words are associated with the input image generation model. The original image is then encoded into image feature information through a VAE encoder.

[0122] The prompt words are encoded into text feature information using a text encoder;

[0123] Image and text feature information are correlated and input into the Unet module for feature mapping processing, and predicted feature information is output.

[0124] The VAE decoder decodes the predicted feature information and outputs the model-generated image.

[0125] Fifthly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions being executed by the processor to implement the steps of the method described in the first aspect, or to implement the steps of the method described in the second aspect.

[0126] In a sixth aspect, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect, or implement the steps of the method described in the second aspect.

[0127] In a seventh aspect, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect, or to implement the steps of the method as described in the second aspect.

[0128] Eighthly, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method as described in the first aspect, or to implement the steps of the method as described in the second aspect.

[0129] In this embodiment, the electronic device determines a sample image set based on an initial sample image set. This sample image set includes at least one sample image. Each sample image includes a key region, which includes a face region and a shoulder region. The width and height of the face region are both greater than or equal to a first threshold. The face region has pore features and no blemishes. Based on the sample image set, the electronic device trains the initial model to obtain an image generation model. This solution constructs a training set by selecting sample images containing face and shoulder regions, and whose skin exhibits both pore features and blemish-free characteristics. The initial model is then trained to obtain the image generation model, thereby enhancing the skin texture of the generated portrait. This improves the realism of the skin texture while effectively avoiding blemish generation, thus improving the quality of the generated image. Attached Figure Description

[0130] Figure 1 A schematic flowchart illustrating a model training method provided for some embodiments of this application;

[0131] Figure 2 A schematic diagram illustrating the user-customized beautification process provided for some embodiments of this application;

[0132] Figure 3 A schematic diagram of the algorithm module in a digital clone model provided for some embodiments of this application;

[0133] Figure 4A Schematic diagram of key regions of images provided for some embodiments of this application;

[0134] Figure 4B Schematic diagram of key regions of images provided for some embodiments of this application;

[0135] Figure 4C Schematic diagram of key regions of images provided for some embodiments of this application;

[0136] Figure 4D Schematic diagram of key regions of images provided for some embodiments of this application;

[0137] Figure 5A A schematic diagram of candidate sample images provided for some embodiments of this application;

[0138] Figure 5B A schematic diagram illustrating the processing flow of a multimodal scoring model for four candidate sample images provided in some embodiments of this application;

[0139] Figure 6 Schematic diagrams of images adjusted to different sizes provided for some embodiments of this application;

[0140] Figure 7A schematic diagram illustrating the multi-resolution collaborative training process provided for some embodiments of this application;

[0141] Figure 8 A schematic diagram of the four main training stages of the model training method provided for some embodiments of this application;

[0142] Figure 9 A schematic diagram illustrating the process of obtaining a complete face image and a partial face image corresponding to each candidate image, provided for some embodiments of this application;

[0143] Figure 10 A schematic diagram illustrating a training process based on global and local graphs, provided for some embodiments of this application;

[0144] Figure 11 Detailed illustrations of the training process for the LoRa module provided for some embodiments of this application.

[0145] Figure 12 A schematic diagram illustrating the visual training process of the initial model provided for some embodiments of this application;

[0146] Figure 13 A flowchart illustrating the construction process of a digital clone model provided for some embodiments of this application;

[0147] Figure 14 A flowchart illustrating an image generation method provided for some embodiments of this application;

[0148] Figure 15 A flowchart illustrating an image generation method provided for some embodiments of this application;

[0149] Figure 16 Schematic diagrams of the structure of the model training apparatus provided for some embodiments of this application;

[0150] Figure 17 Schematic diagrams of the structure of an image generation apparatus provided for some embodiments of this application;

[0151] Figure 18 Schematic diagrams of the structure of electronic devices provided for some embodiments of this application;

[0152] Figure 19 A schematic diagram of the hardware structure of an electronic device provided for some embodiments of this application. Detailed Implementation

[0153] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0154] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0155] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included items. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more items, and its meaning is similar to that of "at least one."

[0156] The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. The terminology involved in the embodiments of this application is explained below.

[0157] Diffusion model: A generative model whose core idea is to gradually recover an image from noise by simulating the diffusion process in physics, thereby generating a high-quality image.

[0158] Human portrait model: A human portrait model is a diffusion model that is specially designed and trained and highly specialized to generate high-quality, high-fidelity human portraits. It is usually based on a latent diffusion model architecture and generates images in a latent space of a pre-trained autoencoder.

[0159] Texture: Texture refers to the visual and tactile characteristics of an image, such as the realism and smoothness of human skin. In scenarios where portraits are generated using portrait models, texture is an evaluation metric for assessing the quality of the generated images.

[0160] Low-Rank Adaptation (LoRA) is an efficient model fine-tuning technique primarily used for lightweight adaptation on large pre-trained models (such as large language models). Its core idea is to reduce the number of parameters required for fine-tuning through low-rank decomposition, thereby reducing computational and storage costs while maintaining model performance.

[0161] Variational Autoencoder (VAE): A VAE is a generative model that combines deep learning and probabilistic graphical models. It addresses the lack of structure in the latent space of traditional autoencoders by introducing variational inference, enabling the generation of continuous and meaningful latent representations for data generation. It consists of two modules: a VAE encoder and a VAE decoder. The encoder transforms an image into latent space features, and the decoder transforms those features back into an image.

[0162] Text Encoder: A text encoder that encodes text prompts into the feature space. It is a conditional injection module in a diffusion model used for text-controlled images.

[0163] With the rapid development of AI technology, the demand for AI-based image generation has gradually emerged. The advent of generative models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) has laid a feasible foundation for this demand. However, GANs and VAEs suffer from training instability and limited generation quality in certain tasks. This has prompted researchers to explore more stable and efficient generation methods, leading to the emergence of diffusion models. The outstanding performance of works such as DDPM (Denoising Diffusion Probabilistic Models), DDIM (Denoising Diffusion Implicit Models), and Stable Diffusion has driven the widespread application of diffusion models in academia and industry. Since portraiture is the application scenario with the greatest demand for image generation, it is essential to develop a model specifically designed to excel at drawing people.

[0164] The execution entity of the model training method provided in this application embodiment can be a model training device, which can be an electronic device, or a functional module or functional entity within an electronic device. The following description uses the execution of the model training method by a model training device as an example to illustrate the model training method provided in this application embodiment.

[0165] The model training method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0166] The model training method provided in this application can be applied to scenarios where training an image generation model is required. In such scenarios, users expect to train an image generation model that can generate highly usable images of people with good skin texture and clear skin details when magnified, in order to meet the application requirements that have high requirements for skin texture performance, such as scenarios that require the generated images to be beautiful and realistic, conforming to the characteristics of the human figure, and requiring the skin texture to have realism and fineness, such as generating magazine covers, artistic portraits, and beauty products.

[0167] Figure 1 This is a flowchart illustrating the model training method provided in the embodiments of this application, as shown below. Figure 1 As shown, the model training method may include the following steps 201 and 202:

[0168] Step 201: The model training device determines the sample image set based on the initial sample image set.

[0169] The aforementioned sample image set includes at least one sample image; each sample image includes a key region, which includes a face region and a shoulder region. The width and height of the face region are both greater than or equal to a first threshold. The face region has pore features and no blemish features.

[0170] In some embodiments of this application, the initial sample image set may include multiple acquired portrait images. For example, approximately 50,000 portrait images with clear skin texture and no obvious acne, blemishes, or other imperfections may be collected as the initial sample image set.

[0171] It should be noted that this initial set of sample images can also be called the candidate training set.

[0172] In some embodiments of this application, the model training device filters the initial sample image set to obtain a sample image set.

[0173] For example, the screening process may include the following steps: In the original image, the key regions containing the face and shoulders are identified and extracted using the DWpose key point detection algorithm. Then, based on the length and width of the key regions, the length is randomly expanded by 1%-5% to obtain a large face image, and the key regions are extracted from the image. Next, a multimodal large model, such as InternVL-2.5, is used to machine-label each image. The image is judged by a question-and-answer method to determine whether the facial skin in the image has clear pores and no blemishes such as acne scars or spots. Only images with clear pores and no blemishes such as acne scars or spots are retained. Then, the face detector is used to locate the face region in the image, and images with a minimum side size of less than 600 pixels are filtered out to ensure sufficient facial details. Finally, approximately 5,000 high-quality portrait images are obtained to form a sample image set.

[0174] It should be noted that the first threshold mentioned above can be set according to actual needs, for example, it can be set to 600px, but this application embodiment does not limit it.

[0175] It should be noted that pore features refer to the visible skin texture and pore structure in the facial skin area.

[0176] It should be noted that imperfections include, but are not limited to, acne scars, blemishes, acne, or other skin defects.

[0177] Step 202: The model training device trains the initial model based on the sample image set to obtain the image generation model.

[0178] In some embodiments of this application, the initial model described above can be a diffusion model or a variant thereof. The core structure of the initial model includes a Unet module, a variational autoencoder, and a text encoder. By training the initial model to optimize its model parameters, the resulting image generation model can generate images with highly realistic human skin texture.

[0179] It should be noted that the Unet module can also be called the Unet network; the variational autoencoder is VAE Encoder; and the text encoder is Text Encoder.

[0180] In some embodiments of this application, the image generation model described above can be a portrait generation model.

[0181] It should be noted that the human portrait generation model can also be called a human portrait model.

[0182] It's important to note that as a model specifically designed for generating portraits, the portrait generation model must simultaneously meet several requirements: high aesthetic appeal, realism, strong text-following ability, and low distortion rate. Aesthetic appeal refers to whether the generated portrait is visually appealing. Realism refers to whether the generated portrait conforms to objective facts, and whether details such as skin and hair resemble those of a real person; the opposite of this is an "AI-like" or "artificial" appearance. Text-following ability refers to whether the image generated by the portrait generation model closely matches the prompts input to it.

[0183] In some embodiments of this application, the initial model described above can be a human portrait base model.

[0184] In some embodiments of this application, the portrait base model can be a portrait base model module in a digital clone model. This digital clone model is a digital clone model used in the creation of a digital clone model for user-customized beautification scenarios.

[0185] Figure 2 This is a schematic diagram of the user-customized beautification process provided in the embodiments of this application, such as... Figure 2 As shown, the process includes the following steps:

[0186] Step 51: The model training device executes the digital clone management process.

[0187] Step 52: The model training device verifies whether the user is logged in. If not logged in, proceed to step 53; if logged in, proceed to step 54.

[0188] Step 53: Log in to the model training device account.

[0189] Step 54: The model training device detects whether a digital clone exists. If it exists, proceed to step 55; otherwise, proceed to step 56.

[0190] Step 55: The model training device displays, deletes, resets, or optimizes digital clones.

[0191] Step 56: The model training device creates a digital clone.

[0192] Step 57: The model training device determines whether a character category already exists. If a character category exists, proceed to steps 58a and 58b. If no character category exists, proceed to step 58.

[0193] Step 58a: The model training device prompts the user to create a digital clone.

[0194] For example, the model training device prompts the user to select a person based on an existing classification and create a digital clone.

[0195] Step 58b: The model training device acquires photos.

[0196] Step 59: The model training device prompts the user to select a photo.

[0197] For example, after step 58a, the following steps may be included:

[0198] Step 60: The model training device determines whether the photo is usable.

[0199] Step 61: The model training device determines whether there are enough available photos. If there are not enough available photos, proceed to step 59; if there are enough available photos, proceed to step 62.

[0200] Step 62: The model training device calls upon the large cloud model or the local large model to create a digital clone model.

[0201] For example, the digital clone model can be managed after it is created.

[0202] For example, the presence of a digital clone can be determined by detecting whether a digital clone is generated from a large cloud model or a local large model.

[0203] For example, digital clones can be created through people clustering, large image swipe-up recommendations, and AI beauty enhancement editing guidance.

[0204] For example, a digital clone model can be created using a large cloud model or by selecting a local large model.

[0205] In some embodiments of this application, the algorithm module in the digital avatar model is divided into three parts: the basic portrait model, the user LoRa, and the beautification module. The post-processing module processes the generated image and is therefore included in the beautification module. The module connection logic is as follows: Figure 3 As shown, the portrait base model 31 is the core module, mainly used to generate realistic user portraits based on the image to be beautified; the user Lora module 32 is mainly used to learn the user's personalized appearance based on the user's diverse life photos, so that the portrait base model can generate the user's face; the post-processing module 33 is used to work with the beautification module 34 to beautify the user image generated by the portrait base model.

[0206] It's important to note that while a basic portrait model alone can generate aesthetically pleasing images with good skin texture, its training data doesn't include user data. Therefore, a single basic portrait model cannot generate a model that perfectly matches the user's appearance. Custom training of user LoRa parameters is necessary, which are then added to the Unet module. User LoRa is essentially an additional learnable linear layer added to the Unet part of the basic portrait model. This allows the basic portrait model to learn new concepts with very few learnable parameters, while maintaining almost no change in its generation performance. In customized beautification scenarios, a series of candidate images are provided through album clustering. Users select at least 10 personal images from these images to train the user LoRa. During application, the user LoRa is integrated into the basic portrait model, enabling it to generate the user's face. However, these two modules alone cannot meet the image beautification requirements. Therefore, a beautification module and a post-processing module need to be combined to work synergistically and produce the final effect. The workflow of several algorithm modules is as follows:

[0207] Step 11: The model training device trains the basic human portrait model.

[0208] Step 12: The model training device trains the beautification module on the basic human portrait model structure.

[0209] For example, both the basic portrait model and the beautification module are trained in the cloud. Users only need to call the cloud model when using it. If the device side is supported, the model needs to be downloaded to the local machine in advance.

[0210] Step 13: The model training device trains user Lora on the human portrait basic model structure.

[0211] It should be noted that, due to limitations of the business scenario, LoRa training can only be performed at a 512×512 resolution.

[0212] Step 14: The model training device integrates user Lora and beautification modules on the basic human portrait model structure to obtain a customized beautification model.

[0213] Step 15: The model training device processes the generated image through the post-processing module to obtain the final beautified image.

[0214] In some embodiments of this application, the model training device inputs a sample image set into the initial model, and through phased multi-resolution collaborative training and loss backpropagation, iteratively optimizes the weight parameters of modules such as Unet in the initial model to obtain an image generation model with high skin texture generation capability.

[0215] In some embodiments of this application, high-quality images can be used as a benchmark to calculate the perceptual loss of low-quality images, i.e., the quality difference, and the quality difference can be used as an additional optimization objective to guide the model to learn the distribution of high-quality data.

[0216] The model training method provided in this application involves a model training device determining a sample image set based on an initial sample image set. This sample image set includes at least one sample image. Each sample image includes a key region, comprising a face region and a shoulder region. The width and height of the face region are both greater than or equal to a first threshold. The face region exhibits pore features and is free of blemishes. The model training device trains the initial model based on the sample image set to obtain an image generation model. This method constructs a training set by selecting sample images containing face and shoulder regions, and whose skin exhibits both pore features and blemish-free characteristics. Training the initial model yields an image generation model, thereby enhancing the skin texture of the generated portrait. This improves the realism of skin texture while effectively avoiding blemish generation, thus enhancing the quality of the images generated by the image generation model.

[0217] In some embodiments of this application, prior to step 201 above, the model training method provided in this application may further include the following step 203:

[0218] Step 203: The model training device acquires the initial sample image set.

[0219] The initial sample image set mentioned above includes at least one candidate sample image;

[0220] For example, in conjunction with step 203 above, step 201 above may include steps 201a to 201e:

[0221] Step 201a: The model training device performs key point detection on each candidate sample image and identifies key regions in each candidate sample image.

[0222] Step 201b: The model training device expands the key regions based on the width and height of the key regions in each candidate sample image, and determines the expanded key regions as the cropping regions.

[0223] Step 201c: The model training device extracts the cropped region of each candidate sample image to obtain a cropped sample image set.

[0224] Step 201d: The model training device inputs the cropped sample image set into the multimodal scoring model and outputs the image quality information of each candidate sample image. The image quality information indicates whether there are pore features in the face region and whether there are blemish features in the face region in each candidate sample image.

[0225] Step 201e: The model training device determines the sample image set based on the image quality information of each candidate sample image.

[0226] In some embodiments of this application, the model training device may use a key point detection algorithm to process the image in order to accurately identify the coordinates of key regions containing the contours of the face and shoulders.

[0227] In some embodiments of this application, the model training device may use the width and height of the key region in each candidate sample image as a reference to randomly expand the key region by a fixed length.

[0228] For example, taking an expansion of 1% to 5% in length as an example, the original key area can be randomly expanded by several pixels. For example, a 512×512 area can be expanded by 5-10 pixels to obtain a 522×532 cropping area, thereby introducing a small amount of surrounding context information while retaining the core portrait information.

[0229] In some embodiments of this application, the model training device crops out the expanded key region from each candidate sample image to obtain a cropped sample image set. In this way, by extracting a uniform image region centered on the human figure from the original image, interference from irrelevant backgrounds on subsequent quality evaluation and model training is avoided, while image distortion in subsequent processing is reduced.

[0230] The following example illustrates the process of obtaining a cropped sample image set.

[0231] For example, since image resizing is required during model training, reducing the resolution of images in the training set directly leads to poor skin texture and an overly AI-like appearance in the generated images, which does not meet the usage scenario. Considering factors such as effect gain, computing power, and time consumption, 1024*1024 resolution is usually the upper limit for the bucket size of pre-trained models. However, in real portrait images, when the face occupies less than 10%, resizing to 1024*1024 resolution results in the face region having fewer than 102*102 pixels, leading to a significant loss of detail. To reduce the quality loss caused by resizing, as... Figure 4A As shown, the model training device first identifies and extracts key regions containing faces and shoulders in the original images of the initial sample image set using the DWpose keypoint detection algorithm. The DWpose detection results are shown in the image below. Figure 4B As shown, the key points of the key area are as follows: Figure 4C As shown, then, based on the length and width of the key region, the length is randomly expanded by 1%-5% to obtain a larger face image, i.e., a cropped sample image, such as... Figure 4D As shown.

[0232] In some embodiments of this application, the model training device inputs a cropped sample image set into a multimodal scoring model and outputs the image quality information of each candidate sample image in a question-and-answer format.

[0233] The following example illustrates the processing procedure of the multimodal scoring model.

[0234] For example, a multimodal large model is used for automatic labeling in the form of image-text question-and-answer. The prompt could be designed as: "You are an experienced high-quality image screener. Please judge whether the pores of the facial skin in the input image are clear, and whether there are blemishes such as spots or acne scars. Answer only yes or no, with the output format being [Yes, No]." If the multimodal scoring model outputs "Yes" or "No," it represents that the image has pore features and the image does not have blemish features, respectively. Images that the multimodal scoring model responds with "Yes" or "No" are the filtered data, and the image processing device can determine the sample image set based on the filtered images. For example, only images whose multimodal scoring model outputs "Yes" or "No," that is, candidate images that simultaneously satisfy "having pore features" and "not having blemish features," are retained to constitute the final high-quality sample image set used for training. For example, this multimodal large model can be InternVL-2.5.

[0235] In this embodiment, by cropping and expanding the key regions of the image, the model can better learn the real texture features of key regions such as skin regions. Through multimodal modeling, high-quality face images with both pore details and flawless features are accurately selected, thereby providing high-quality sample images for model training. This enables the model to generate images with realistic skin texture and clear details.

[0236] In some embodiments of this application, step 201e above may include step 201e1:

[0237] Step 201e1: The model training device determines the sample image set by all candidate sample images that meet the first condition.

[0238] The first condition includes: the image quality information indicates that the face region of the candidate sample image has pore features and the face region does not have blemish features, and the width and height of the face region of the candidate sample image are both greater than or equal to the first threshold.

[0239] In some embodiments of this application, the model training device can use the image quality information output by the multimodal scoring model to indicate that the face region has pore features and no blemish features as candidate sample images that have passed the screening, and determine the images in the candidate sample images in which the width and height of the face region are both greater than or equal to a first threshold as the sample image set.

[0240] For example, combining the above example, the image that the multimodal scoring model responds with "yes" or "no" is the candidate sample image that passes the screening. Then, the face detector is used to obtain the coordinates of the face bounding box, and the width and height of the face bounding box are calculated. Candidate sample images with a minimum side of face bounding box less than 600 pixels are filtered out. Figure 5A As shown, the face region has a pixel width and height of 618×609, while the original image has a width and height of 1000×1333 pixels. When either the width or height of the face region is less than 600 pixels, the image quality is considered poor, affecting the training effect of the model. Therefore, such images are filtered out. Figure 5B As shown, after inputting four candidate sample images (Image 1, Image 2, Image 3, and Image 4) into the multimodal scoring model, the model evaluates whether the skin pores in each candidate sample image are clear, whether the pores are visible, and whether there are blemishes such as spots in each candidate sample image. Subsequently, face resolution detection is performed on the images that pass the semantic screening, requiring that the face region size is not lower than a first threshold. Only images that simultaneously meet the criteria of "clear pores," "no blemishes," and "meeting the face resolution standard" are ultimately retained. If any condition is not met, the image is filtered out, meaning that Images 1 and 4 are ultimately retained. In this way, by fusing image feature information and pixel size requirements, sample images with both high-quality skin texture and sufficient detail can be efficiently selected.

[0241] It should be noted that, Figure 5B In the image, a "√" indicates that the face resolution meets the standard, and an "×" indicates that the image does not meet any of the conditions.

[0242] It should be noted that, Figure 5A The dashed box area is used to indicate the face area, which is 618×609 in size.

[0243] In this embodiment, by performing dual filtering on candidate sample images, that is, simultaneously meeting the requirements for skin quality and face resolution, images that are too small and therefore have blurred details and cannot provide effective texture information are effectively filtered out. Images that have clear pores and no flaws, as well as high resolution, are selected. This ensures that the final sample image set has both high-quality skin texture and sufficient effective pixel information, so that the trained image generation model can generate images with realistic skin texture and rich details.

[0244] In some embodiments of this application, prior to step 202 above, the model training method provided in this application may further include steps 204 and 205:

[0245] Step 204: The model training device scales and crops each sample image in the sample image set to generate a first-size image, a second-size image, and a third-size image corresponding to each sample image. The image size of the first-size image is smaller than that of the second-size image, and the image size of the second-size image is smaller than that of the third-size image.

[0246] Step 205: The model training device determines all images of the first size as the first sample image set, all images of the second size as the second sample image set, and all images of the third size as the third sample image set;

[0247] For example, in conjunction with steps 204 and 205 above, step 202 above may include the following step 202b:

[0248] Step 202b: The model training device trains the initial model based on the first sample image set, the second sample image set, and the third sample image set.

[0249] In some embodiments of this application, the image size of the first-size image described above can be set to 768×768 pixels.

[0250] In some embodiments of this application, the image size of the second-size image described above can be set to 1024×1024 pixels.

[0251] In some embodiments of this application, the image size of the third-size image described above can be set to 1280×1280 pixels.

[0252] In some embodiments of this application, for each sample image in the sample image set, its shorter side is first scaled to the target size, such as 1280 pixels, while maintaining the aspect ratio of the original image; then, the image is cropped according to the shorter side to obtain a standard square image. This process is repeated to generate images of three sizes with shorter sides of 1280, 1024, and 768 pixels, respectively, thereby forming sample image sets of high, medium, and low resolutions, namely the third sample image set, the second sample image set, and the first sample image set.

[0253] For example, each sample image is resized to 1280, 1024, or 768 along its shorter side, maintaining the same aspect ratio as the original image. For instance, if the original image is 4096×2048 and you want to resize it to 1024×1024, you first resize it to 2048×1024. Then, using the center point of the longer side as the origin, you crop the length of the longer side that exceeds the shorter side. In this example, you crop 512 pixels from each side, ultimately obtaining 1024×1024 resolution data. Figure 6As shown, from left to right, images 61, 62, and 63 are schematic diagrams illustrating the effects of scaling the images to 1280, 1024, and 768 by the shorter side, respectively. The area marked by the dashed box in each image represents the symmetrical cropping from the center of the longer side as the origin of the coordinate system to both sides of the longer side. The length to be retained on both sides is equal to half the length of the shorter side. The area within the dashed box is the effective part to be retained, and the rest will be cropped.

[0254] It should be noted that the image width in this example is smaller than its height, so the width is the shorter side.

[0255] In some embodiments of this application, the model training device adopts a phased collaborative training strategy. First, a second sample image set is used as a medium-resolution input for preliminary training to adapt the model to the data distribution. Then, a third sample image set is introduced as a high-resolution input to learn more refined skin texture details. Next, the first sample image set is used as a low-resolution input, and by combining it with the high-resolution third sample image set, the model can still generate high-quality skin details under low-resolution conditions.

[0256] It should be noted that high-resolution images can provide rich detailed features for the initial model, while low-resolution images can simulate the limited scenarios in real-world applications. This application combines images of multiple resolutions for model training, enabling the model to effectively transfer high-resolution detailed information to the low-resolution generation process, thereby avoiding the loss of image details during the resolution model inference process.

[0257] In this embodiment, by constructing a multi-resolution sample set and performing co-training, the initial model can simultaneously learn image features of different sizes from low resolution to high resolution, thereby enhancing its generation robustness at various resolutions and enabling it to output high-quality generated images during low-resolution inference.

[0258] In some embodiments of this application, step 202 may include steps 202c and 202d:

[0259] Step 202c: The model training device inputs all second-size images from the second sample image set into the initial model, and performs feature mapping processing on each second-size image through the Unet module of the initial module to obtain the first predicted feature information of each second-size image;

[0260] Step 202d: The model training device updates the model parameters of the Unet module based on the first difference information between the first predicted feature information of each second-size image and the reference feature information of each second-size image;

[0261] The reference feature information for each second-size image is obtained by performing latent space feature mapping on each second-size image through the VAE encoder of the image generation module.

[0262] In some embodiments of this application, the second sample image set described above may consist of sample images with a resolution of 1024×1024 pixels.

[0263] In some embodiments of this application, the first predicted feature information and the reference feature information mentioned above are both feature vectors.

[0264] In some embodiments of this application, the model training device calculates the latent space representation of the input second-size image through the Unet module to extract deep features of the sample image.

[0265] It should be noted that this Unet module can become the main Unet module.

[0266] In some embodiments of this application, the reference feature information of the second-size image described above can be represented by latent space.

[0267] In some embodiments of this application, the model training device performs latent space feature mapping on each second-size image through a VAE encoder, converting each second-size image into a structured feature representation in the latent space, i.e., reference feature information.

[0268] In some embodiments of this application, the aforementioned first difference information is used to measure the distance between the first predicted feature information and the reference feature information, and can be calculated using loss functions such as mean squared error or L1 Loss.

[0269] The following is an illustrative example of the process of training the initial model based on the second set of sample images.

[0270] For example, the model training device inputs all second-size images into the initial model, converts each second-size image into a feature vector in the latent space through a VAE encoder, and uses it as reference feature information. Then, it inputs the feature vector of each second-size image into the Unet module for feature mapping processing and outputs its predicted feature vector, i.e., the first predicted feature information. The model training device uses the error between the reference feature information and the predicted feature information of the second-size image as the first difference information, i.e., the loss value, and updates the model parameters of the Unet module through the backpropagation algorithm, so that the predicted features output by the Unet module continuously approach the high-quality feature vector generated by the VAE encoder.

[0271] In this embodiment of the application, the initial model is trained by using medium-resolution image samples, so that the initial model can effectively learn the image features contained in the medium-resolution images while adapting to the training data, thereby enhancing the model's ability to perceive and extract details.

[0272] In some embodiments of this application, after step 202d above, the model training method provided in this application embodiment may further include the following steps 206 and 207:

[0273] Step 206: The model training device inputs at least one third-size image from the third sample image set into the image generation model, and performs feature mapping processing on each third-size image through the Unet module to obtain the second predicted feature information of each third-size image.

[0274] Step 207: The model training device updates the model parameters of the Unet module based on the second difference information between the second predicted feature information and the reference feature information of the third-size image;

[0275] The reference feature information for each third-size image is obtained by performing latent space feature mapping on each third-size image through the VAE encoder of the image generation module.

[0276] In some embodiments of this application, the aforementioned third sample image set may consist of sample images with a resolution of 1280×1280 pixels.

[0277] In some embodiments of this application, the second predicted feature information and the reference feature information mentioned above are both feature vectors.

[0278] In some embodiments of this application, the model training device calculates the latent space representation of the input third-size image through the Unet module to extract deep features of the sample image.

[0279] In some embodiments of this application, the reference feature information of the third-size image described above can be represented by latent space.

[0280] In some embodiments of this application, the model training device performs latent space feature mapping on each third-size image through a VAE encoder, converting each third-size image into a structured feature representation in the latent space, i.e., reference feature information.

[0281] In some embodiments of this application, the aforementioned second difference information is used to measure the distance between the second predicted feature information and the reference feature information, and can be calculated using loss functions such as mean squared error or L1 Loss.

[0282] The following is an illustrative example of the process of training the initial model based on a third set of sample images.

[0283] For example, the model training device inputs all third-size images into an initial model, which can be an initial model trained based on a second set of sample images; each third-size image is converted into a feature vector in the latent space by a VAE encoder as reference feature information; then, the feature vector of each third-size image is input into the Unet module for feature mapping processing, and its predicted feature vector is output, i.e., the second predicted feature information; the model training device uses the error between the reference feature information and the corresponding predicted feature information of the third-size image as the second difference information, i.e., the loss value, and updates the model parameters of the Unet module through the backpropagation algorithm, so that the predicted features output by the Unet module continuously approach the high-quality feature vector generated by the VAE encoder.

[0284] In this embodiment, the initial model is first trained using medium-resolution images to adapt to the training data. Then, high-resolution image samples are used to train the initial model, enabling it to learn the richer image features contained in the high-resolution images, thereby further enhancing the model's ability to perceive and extract details.

[0285] In some embodiments of this application, after step 207 described above, the model training method provided in this application may further include steps 208 to 212:

[0286] Step 208: The model training device constructs a reference Unet module based on the updated model parameters of the Unet module;

[0287] Step 209: The model training device inputs the reference feature information of the third-size image corresponding to each sample image into the reference Unet module and outputs the third predicted feature information;

[0288] Step 210: The model training device fuses the reference feature information of the first-size image and the reference feature information of the third-size image corresponding to each sample image, and outputs the fused feature information.

[0289] Step 211: The model training device inputs the fused feature information into the Unet module and outputs the fourth predicted feature information;

[0290] Step 212: The model training device updates the weight parameters of the Unet module based on the difference between the third and fourth predicted feature information.

[0291] In some embodiments of this application, the aforementioned reference Unet module can be a Unet module constructed based on the weight parameters of an updated Unet module. The weights of this reference Unet are frozen and do not participate in subsequent training processes, serving to provide a high-quality feature reference.

[0292] In some embodiments of this application, the model training device performs latent feature mapping processing on the third-size image corresponding to each sample image through a VAE encoder to obtain its corresponding reference feature information, i.e., latent space representation, and inputs the reference feature information into the reference Unet module; then, the reference Unet performs forward calculation on the reference feature and outputs predicted feature information, i.e., third predicted feature information.

[0293] In some embodiments of this application, the model training device fuses the reference feature information of the first-size image corresponding to the same sample image with the reference feature information of the third-size image. For example, the reference feature information of the first-size image and the reference feature information of the third-size image are concatenated or added in the dimension of the feature token to generate fused feature information, thereby obtaining fused feature information that combines high-resolution detail information and low-resolution context information.

[0294] In some embodiments of this application, the model training device inputs the fused feature information into the Unet module, performs forward computation, and outputs the fourth predicted feature information; then it calculates the difference information between the third and fourth predicted feature information, and updates the model parameters of the Unet module based on the difference information, so that its output is close to the output of the reference Unet.

[0295] In some examples, it can be done through Evaluate the loss between high-quality features and predicted features. The expression is shown in formula (1):

[0296]

[0297] in, For the output of Reference Unet, ε θ This is the output of Unet, and z t The latent space features corresponding to low resolution. Here, t represents the latent space features corresponding to High resolution, t represents the current training time step, and c represents additional cue words as input conditions.

[0298] The following is an illustrative example of the process of jointly training an initial model based on a first-size image and a third-size image.

[0299] For example, assume the third-size image is 1280×1280 and the first-size image is 768×768. First, the frozen reference Unet processes the latent features of the 1280×1280 image, outputting high-quality features, i.e., the third predicted feature information; then, the latent features of the 768x768 image and the latent features of the 1280×1280 image are added and fused along the feature dimension to obtain the fused features; then, the fused features are input into the Unet module to obtain its predicted features, i.e., the fourth predicted feature information; finally, through... Evaluate the loss between high-quality features and predicted features, and update the weights of Unet using this loss. This allows Unet to generate high-quality features from low-resolution inputs that incorporate high-resolution details, similar to those obtained by directly processing high-resolution images.

[0300] Based on the examples above, Figure 7 This is a schematic diagram illustrating the multi-resolution collaborative training process provided in this application. Figure 7 As shown, a high-resolution image with clear skin texture and a low-resolution image of the same person with heavily blurred skin are converted into latent space features by a VAE encoder, yielding High-resolution Tokens and Low-resolution Tokens, respectively. The high-resolution features are input to a frozen reference Unet, whose output serves as a high-quality reference standard. The high- and low-resolution features are fused and interact in the Self-Attention layer of the Unet module, and then processed through a consistency loss function. The output of the main Unet is constrained so that the generated result retains the low-resolution input structure while capturing high-resolution skin details, and finally reconstructs a high-quality portrait image with clear skin texture through the VAE decoder.

[0301] It should be noted that VAE encoder is VAE-Encoder; VAE decoder is VAE-Decoder; reference Unet is reference Unet; High resolution Tokens are high-resolution latent vector representations; Low resolution Tokens are low-resolution latent vector representations; and Self attention is a self-attention mechanism.

[0302] In this embodiment of the application, through collaborative training of images of multiple resolutions, the rich detail information of high-resolution images can be integrated into the processing flow of low-resolution images, thereby avoiding the problem of detail loss caused by low input resolution. This enables the model to learn to generate a high-quality image with rich details of 1280x1280 based on a low-resolution input such as 768x768.

[0303] In some embodiments of this application, after step 212 above, the model training method provided in this application may further include steps 213 to 216:

[0304] Step 213: The model training device constructs at least one first image sample pair based on the first-size image and the second-size image corresponding to each sample image, and constructs at least one second image sample pair based on the second-size image and the third-size image corresponding to each sample image.

[0305] Step 214: The model training device inputs at least one first image sample pair into the initial model, and performs feature mapping processing on the at least one first image sample pair through the Unet module, outputting the fifth predicted feature information.

[0306] Step 215: The model training device inputs at least one second image sample pair into the initial model, and performs feature mapping processing on at least one second image sample pair through the reference Unet module, and outputs the sixth predicted feature information.

[0307] Step 216: The model training device updates the model parameters of the Unet module based on the difference information between the fifth and sixth predicted feature information.

[0308] In some embodiments of this application, the aforementioned first image sample pair consists of a first-size image and a second-size image of the same sample image. The model learns a mapping of detailed features from low resolution to medium resolution using this first image sample pair.

[0309] In some embodiments of this application, the aforementioned second image sample pair consists of a second-size image and a third-size image of the same sample image. This second image sample pair is used by the model to learn detail feature mappings from medium resolution to high resolution.

[0310] In some embodiments of this application, at least one first image sample pair is input into the initial model and converted into a latent feature representation by a VAE encoder; then the latent feature representation is input into the Unet module for feature mapping processing, and the fifth predicted feature information is output.

[0311] In some embodiments of this application, at least one second image sample pair is input into the initial model and converted into a latent feature representation by a VAE encoder; then the latent feature representation is input into the Unet module for feature mapping processing, and the sixth predicted feature information is output.

[0312] In some embodiments of this application, the model training device calculates the difference information between the fifth predicted feature information and the sixth predicted feature information, and updates the model parameters of the Unet module through the backpropagation algorithm based on the difference information, so that its output when processing lower resolution images can be closer to the output of the reference Unet module when processing higher resolution images.

[0313] The following is an illustrative example of the process of jointly training the initial model based on the first image sample pair and the second image sample pair.

[0314] For example, based on images with resolutions of 768×768, 1024×1024, and 1280×1280 corresponding to each sample image, at least one first image sample pair and at least one second image sample pair are constructed. Each first image sample pair includes a 768×768 resolution image and a 1024×1024 resolution image corresponding to the same sample image, and each second image sample pair includes a 1024×1024 resolution image and a 1280×1280 resolution image corresponding to the same sample image. Next, at least one first image sample pair is input into the Unet module, which outputs the predicted feature information corresponding to each image sample pair, i.e., the fifth predicted feature information. At least one second image sample pair is input into a reference Unet module, which outputs the predicted feature information corresponding to each image sample pair, i.e., the sixth predicted feature information. Then, the difference information between the fifth and sixth predicted feature information is calculated. Based on this difference information, the weights of Unet are updated only through backpropagation. In this way, Unet learns to generate feature information that is similar to and rich in features when inputting medium and high resolution inputs, based on low and medium resolution inputs.

[0315] In this embodiment, by training the model using image samples of different resolutions, the model achieves the goal of guiding the low-resolution image processing process with high-resolution information, optimizing the learning direction of the model parameters. This allows the model's feature output when processing "low-medium" resolution image pairs to be close to the feature output when processing "medium-high" resolution image pairs, thereby ensuring that the model can generate rich skin textures and qualities under various input resolutions, and thus improving the quality of the images generated by the model.

[0316] Based on the examples above, Figure 8 A schematic diagram illustrating the four main training stages of the model training method provided in the embodiments of this application, as shown below. Figure 8As shown, the first stage is training based on medium-resolution sample images with a width and height of 1024; the second stage is training based on high-resolution sample images with a width and height of 1280; the third stage is joint training based on high-resolution sample images with a width and height of 1280 and low-resolution sample images with a width and height of 768; and the fourth stage is joint training based on a first image sample pair consisting of a high-resolution sample image with a width and height of 1280 and a low-resolution sample image with a width and height of 768, and a second image sample pair consisting of a high-resolution sample image with a width and height of 1024 and a low-resolution sample image with a width and height of 768.

[0317] For example, in conjunction with the above Figure 8 Based on the above embodiments, the overall training process, including the training processes of each of the above stages, will be described below by way of example.

[0318] For example, the overall training process includes the following steps:

[0319] Step 21: Resize the original image to 1280, 1024, and 768 pixels along the shorter side, maintaining the same aspect ratio as the original. For example, if the original image is 4096*2048 and you want to resize it to 1024*1024, first resize it to 2048*1024. Then, using the center point of the longer side as the coordinate center point, crop the length of the longer side that exceeds the shorter side on both sides. In this example, 512 pixels are cropped from each side, resulting in a final resolution of 1024*1024. See the diagram above. Figure 6 As shown, the area enclosed by the dashed box in each image is defined by taking the center point of the longer side as the origin of the coordinate system, and extending upwards and downwards by 1 / 2 * the length of the shorter side by 1 pixel. The area within the dashed box is the valid part that is retained, while the rest will be cropped out.

[0320] Step 22: First, train the basic human portrait model using the commonly used highest training resolution w,h = 1024,1024, to adapt the model to our training data.

[0321] Step 23: Then increase the training resolution to 1280, allowing the Unet model to learn the details of human skin at a higher resolution.

[0322] Step 24: Copy the Unet training weights to the Reference Uet. During training, freeze the Reference Uet weights and only update the Unet weights. Use a 1280 resolution high-resolution input image and a 768 resolution low-resolution input image. The high-resolution and low-resolution images are converted into latent space features by the VAE encoder. The feature dimensions are (batch, num, dim), where batch is the feature data seen in one iteration of model training (there are several feature data for each image), num is the number of feature tokens, and dim is the dimension of each feature token.

[0323] Step 25: Add and fuse the High resolution and Low resolution directly in the num dimension, and feed the fused features into Unet for training. The two input features overlap and interact with each other.

[0324] Step 26: Directly feed the latent space features of the High Resolution layer after VAE encoding into the reference Unet. Inject the results of each layer into the Unet by block, and then connect the output of the reference Unet and the output of the Unet. The consistency is assessed, with the output of Reference Unet serving as the benchmark. The output of Unet needs to gradually approach the output of Reference Unet.

[0325] Step 27: Use two data pairs, one with High Resolution of 1024*1024 and Low Resolution of 768×768, and the other with High Resolution of 1280×1280 and Low Resolution of 768×768, to feed them into Unet and then into Reference Unet to complete the training.

[0326] It should be noted that in the training scenario for the customized beautification user Lora, Lora can only be trained on images with a resolution of 512×512. Therefore, the portrait model needs to be closer to the usage scenario of training and inference at a resolution of 512. Thus, in the third and fourth stages, images with resolutions of 768 and 768 are added for joint training. Here, 768 is chosen instead of 512 as the lower limit of resolution, which ensures that the 512 resolution can be used normally while minimizing the degradation of training data quality. Experimental results show that the model training method provided in this application embodiment can significantly improve the skin texture of portraits, making the generated portraits richer in detail and closer to realistic images.

[0327] In some embodiments of this application, prior to step 202 above, the model training method provided in this application may further include steps 217 and 218:

[0328] Step 217: The model training device acquires at least one candidate image from the sample image set whose face region size is greater than the second size and whose face region proportion is greater than the second threshold.

[0329] Step 218: The model training device acquires the complete face image and the partial face image corresponding to each candidate image.

[0330] For example, in combination with steps 217 and 218 above, step 202 above may include the following step 202d:

[0331] Step 202d: The model training device trains the initial model based on the complete face image and partial face image corresponding to each candidate image.

[0332] In some embodiments of this application, the second size can be 1024×1024 pixels. By selecting images from the sample image set whose face region size is larger than the second size, it is ensured that the face regions in the selected images for training have clear skin texture.

[0333] In some embodiments of this application, the second threshold can be set to 50%. By selecting images from the sample image set whose face region occupies more than the second threshold, it is ensured that the face occupies the main part of the whole image, with fewer background and other interfering elements, so that the model can focus on learning facial features.

[0334] For example, the model training device can select 1,000 candidate images from the sample image set with a face resolution greater than 1024*1024 and a face proportion greater than 50%.

[0335] In some embodiments of this application, for each candidate image, a complete face image and a partial face image are obtained.

[0336] It should be noted that the complete face image is an image of the face region from the candidate images, containing global facial feature information; the partial face image only contains the eye region, mouth region, nose region, etc., avoiding interference from hair, background, etc.

[0337] For example, assuming the sample image set includes 5000 sample images, the model training device selects approximately 1000 candidate images from these 5000 sample images, where the face resolution is greater than 1024x1024 and the face occupies more than 50% of the image. For each candidate image, the face region is first accurately located using a face detection and segmentation model, and the face region is cropped to obtain a complete face image. Then, multiple 768x768 pixel partial face images are randomly cropped from the complete face image. Finally, during training, a complete face image and at least one partial face image corresponding to each candidate image are combined to form a face sample set.

[0338] In some embodiments of this application, the model training device inputs the complete face image and the corresponding local face image into the initial model for training, so as to learn to reconstruct the complete global portrait and local texture features.

[0339] In this embodiment, by cropping the sample image to obtain a complete face image and a partial face image, the model can learn the overall composition of the face image based on the complete face image, and learn the detailed features of the skin pores, texture and other features of the face based on the partial face image, thereby enabling the model to generate more realistic and high-quality face images.

[0340] It should be noted that step 202d can also be performed after step 216. The execution order of step 202d is not limited in this embodiment.

[0341] In some embodiments of this application, step 218 described above can be implemented by steps 218a to 218e:

[0342] Step 218a: The model training device inputs each candidate image into the face detection model and outputs the facial feature point information of each candidate image;

[0343] Step 218b: The model training device performs face cropping on each candidate image based on the facial feature point information of each candidate image to obtain the complete face image corresponding to each candidate image;

[0344] Step 218c: The model training device inputs the complete face image corresponding to each candidate image into the fine segmentation model and outputs a binary mask image corresponding to each complete face image. The image pixel values ​​of the binary mask image include white area pixel values ​​and black area pixel values. The image region corresponding to the white area pixel values ​​is the face skin region, and the image region corresponding to the black area pixel values ​​is the non-face skin region.

[0345] Step 218d: The model training device determines the image region in all complete face images that corresponds to the mask image region that satisfies the second condition as the sampling region; wherein, the second condition is that the image pixel value of the mask image region in the binary mask image is the white area pixel value;

[0346] Step 218e: The model training device crops the sampled area of ​​each complete face image according to the reference cropping size, and determines the cropped image as the local face image corresponding to each candidate image.

[0347] In some embodiments of this application, facial feature point information can be a set of coordinate points that mark key parts of the face, identified by a face detection model. The location and extent of the face can be accurately located through facial feature point information.

[0348] For example, the face detection model can be DWpose or RetinaFace, etc.

[0349] For example, key facial features can be the corners of the eyes, the tip of the nose, the corners of the mouth, the facial contours, etc.

[0350] For example, a candidate image is input into a pre-trained face detection model, which outputs an array containing the coordinates of multiple feature points, which outline the face contour and the position of facial features in the image.

[0351] In some embodiments of this application, the model training device calculates the smallest region containing all facial feature points in a candidate image based on feature point coordinates. The candidate image is then cropped according to this region to obtain a complete face image.

[0352] In some embodiments of this application, the refined segmentation model can be the Face Parsing model.

[0353] It should be noted that the refined segmentation model is a pixel-level classification model that can divide each pixel in an image into predefined categories such as skin, eyebrows, eyes, lips, and background.

[0354] In some embodiments of this application, the binary mask image contains only two types of pixel values: white area pixel values ​​and black area pixel values. The white area pixel value can be 1 or 255, corresponding to the skin area of ​​a human face; the black area pixel value is usually 0, corresponding to the non-skin area of ​​a human face.

[0355] Understandably, non-facial skin areas refer to areas other than the facial skin area, such as hair and background.

[0356] It should be noted that the above binary mask image can be called a mask image.

[0357] For example, the cropped complete face image is input into the Face Parsing model, which outputs a black and white mask image of the same size as the original image, in which the skin areas such as cheeks, forehead, and nose are white, and the hair and all background areas are black.

[0358] In some embodiments of this application, the model training device program traverses each pixel of the mask image. For the pixel currently being processed, it reads its pixel value to obtain the grayscale value of the pixel in the mask image. Then, it checks whether the pixel value is equal to 1 or 222. If the current pixel value is equal to 255, it determines that the position corresponding to the pixel belongs to a valid facial skin region. Then, it records the coordinate position of the pixel in the mask image coordinate system and performs coordinate mapping based on the coordinate position and the original complete face image to determine the coordinate position corresponding to the coordinate position in the original complete face image. Finally, it determines the sampling area in the original complete face image using the recorded set of pixel coordinates.

[0359] It is understandable that the binary mask image and the original complete face image are perfectly aligned in space, that is, there is a one-to-one correspondence between pixels. Therefore, the pixel with coordinates (x, y) on the binary mask image corresponds to the pixel with the same coordinates (x, y) in the original complete face image.

[0360] In some embodiments of this application, the aforementioned reference cropping size can be a preset size, such as 768×768 pixels. The model training device randomly slides a window of the reference cropping size within the sampling area and performs one or more cropping operations to obtain one or more partial facial images.

[0361] For example, within a white skin area, a center point is randomly selected, and a 768x768 pixel image region is cropped from this center, thereby obtaining a high-definition local close-up image with rich details and no flaws or interference from a complete face image for subsequent training.

[0362] For ease of description, the complete face image mentioned above can be called a global image, and the partial face image can be called a local image.

[0363] The following example illustrates the process of obtaining the complete face image and partial face image corresponding to each candidate image. For example, as shown... Figure 9 As shown, the process may include the following steps:

[0364] Step 41: Based on the face detection model, obtain the global image 72, i.e., the complete face image, from the candidate image 71;

[0365] Step 42: Then feed the global image into the face refinement segmentation model and output face mask73;

[0366] Step 43: Based on the face mask, lock the value range 74 of the local sampling window;

[0367] Step 44: Within the value range, randomly slide and crop a 768×768 local image to obtain local images 75, 76 and 77;

[0368] It should be noted that, without adjusting or reducing image quality, the model learns the realistic skin texture structure of a person in a local image by adding 'realistic skin texture' to the prompt words for concept binding. In the reasoning stage, only the text label 'realistic skin texture' needs to be input to obtain high-quality human skin texture.

[0369] Step 35: Simultaneously input a complete adult face as the global image, also label it with 'Real Skin Texture', and add 'Complete Face' and 'Partial Face' labels to distinguish between the two.

[0370] It should be noted that this step is to prevent local images from disrupting the overall composition of the portrait, thus preventing the model from automatically generating local images. The global image is the complete face image, while the local image is an image containing only a portion of the face, such as only an eye or mouth. The sampling result for the local image is randomized; it is obtained by randomly sliding a 768×768 pixel bounding box across the face region. The mask image is a black and white image output by the refined face segmentation model, with the facial area in white and the rest, including the hair area, in black.

[0371] In this embodiment, a binary mask is generated by a refined segmentation model, and sampling is performed on the skin region to eliminate interference from non-skin regions such as hair and background. This allows the model to focus on learning skin texture features, thereby generating high-quality images with realistic skin texture. At the same time, by randomly sampling the face region, the model can learn the skin texture features of different parts of the same person, thereby generating a more natural skin texture and improving the image generation effect of the model.

[0372] In some embodiments of this application, step 202d may include steps 202d1 to 202d3:

[0373] Step 202d1: The model training device constructs at least one set of face images based on the complete face image and the partial face image corresponding to each candidate image.

[0374] Step 202d2: The model training device inputs at least one set of face images into the initial model and outputs predicted face feature information.

[0375] Step 202d3: The model training device updates the model parameters of the initial model based on the difference information between the predicted face feature information and the feature information of all complete face images.

[0376] In some embodiments of this application, each set of face images includes a complete face image and at least one partial face image corresponding to the complete face image.

[0377] Understandably, a complete face image provides global structure, composition, and contextual information about the face; a partial face image is an unscaled, ultra-high-resolution close-up of the skin region cropped from the complete image, providing rich details of local skin texture.

[0378] For example, for a candidate image, three 768×768 pixel local images are randomly cropped from its sampling area. Each local image is combined with the original global complete image to form a set of face images for subsequent training.

[0379] In some embodiments of this application, at least one set of pre-constructed face images are input into an initial model, and predicted face feature information is output.

[0380] For example, multiple sets of face images in a batch are input into the initial model. The global and local images in each set of face images are mapped to the latent space by the VAE encoder, and then input together into Unet for feature processing to output predicted face feature information.

[0381] In some embodiments of this application, the feature information of all complete face images refers to the high-quality latent feature representation in the latent space obtained after all complete face images are passed through a VAE encoder.

[0382] In some embodiments of this application, the model training device uses a loss function to calculate the difference between the predicted facial feature information and the reference features, and updates the model's weights based on the difference information. This allows the model to generate subtle texture information in the local image while maintaining the accurate composition of the global image when reconstructing the image.

[0383] Figure 10 This is a schematic diagram of the training process based on global and local graphs, such as... Figure 10As shown, the global image 50 and its corresponding local images 51, 52, and 53 are simultaneously input into the portrait base model. Each image is converted into latent features by a VAE encoder and then fed into the UNet of the portrait base model for training. Within the UNet, an attention mechanism (Attention Map) is used to fuse and enhance features of the global structure and local details, enabling the model to simultaneously learn the overall portrait composition and realistic skin texture details. Finally, the VAE decoder reconstructs a portrait image with a complete composition and high-quality skin. In this way, the model learns the complete composition of a realistic portrait and enhances the information at corresponding locations in the global image with high-resolution local blocks. This allows the model to maintain a complete composition while learning the realistic skin texture in local blocks that is not lost due to size adjustments.

[0384] In this embodiment of the application, by combining the global map and the local map as joint input, the model is trained to process both types of information simultaneously. This allows the model to learn the generation of skin texture details from the local map and the overall facial structure and composition from the global map, thereby improving the visual realism and texture of the generated portrait.

[0385] In some embodiments of this application, step 202 above may include steps 202e1 to 202e4:

[0386] Step 202e1: The model training device adjusts the image size of each candidate image to the first size to obtain a first quality image, blurs the first quality image, and adjusts the image size of the blurred first quality image to the second size and then the first size in sequence to obtain a second quality image;

[0387] The first dimension is larger than the second dimension.

[0388] Step 202e2: The model training device establishes the association between the first quality image and the first image detail cue, establishes the association between the second quality image and the second image detail cue, and constructs at least one image sample pair based on the first quality image and the second quality image.

[0389] The first image detail cue and the second image detail cue indicate the image detail features.

[0390] Step 202e3: The model training device generates a model from at least one image sample pair of the input image, and performs feature mapping processing on at least one image sample pair through the Unet module to obtain the seventh prediction feature information of the first quality image and the eighth prediction feature information of the second quality image.

[0391] Step 202e3: The model training device updates the model parameters of the initial model based on the difference information between the seventh and eighth predicted feature information.

[0392] In some embodiments of this application, the model training device updates the model parameters of the detail enhancement LoRa module of the initial model based on the difference information between the seventh and eighth predicted feature information.

[0393] In some embodiments of this application, the aforementioned detail-enhancing Lora module can be a lightweight adapter module attached to the Unet module. For example, the Unet module may include the detail-enhancing Lora module.

[0394] In some embodiments of this application, the first size can be 1280×1280. For example, the model training device adjusts the image size of the candidate image to the first size of 1280×1280 to preserve rich skin details.

[0395] In some embodiments of this application, the second size can be 384×384. For example, the model training device first applies Gaussian blur to the first quality image to simulate loss of detail; then, the size of the blurred image is sequentially adjusted to 384×384, and then adjusted back to 1280×1280 to degrade the image quality, resulting in a low-quality image with blurred skin texture and loss of detail.

[0396] In some embodiments of this application, the model training device constructs a first image detail cue word for a first quality image, such as rich detail, and labels it as high quality; and constructs a second image detail cue word for a second quality image, such as smooth or blurry detail, and labels it as low quality.

[0397] It is understandable that the first quality image and the second quality image corresponding to the same candidate image have the same content, but there is a significant difference in the richness of detail.

[0398] In some embodiments of this application, the image generation model training device inputs at least one image sample pair into the initial model; after performing latent feature mapping on the first quality image through the VAE encoder, the obtained latent feature representation is input into the Unet module for feature mapping processing, and the seventh predicted feature information is output; and after performing latent feature mapping on the second quality image through the VAE encoder, the obtained latent feature representation is input into the Unet module for feature mapping processing, and the eighth predicted feature information is output.

[0399] In some embodiments of this application, the image generation model training device calculates the difference information between the eighth predicted feature information and the seventh predicted feature information, and updates the model parameters based on the difference information.

[0400] Understandably, the difference between the seventh and eighth predicted feature information represents the difference in processing results of the initial model on images of different quality corresponding to the same image. Based on this difference information, the model parameters of the Unet module are updated, enabling the model to learn to distinguish between text instructions with rich detail and those with smooth detail, as well as to learn the ability to generate images that are close to those with rich detail.

[0401] It's important to note that LoRa training requires a 512 resolution, which can lower the skin texture of portraits. Therefore, a detail enhancement LoRa module is provided, which can be used decoupled from the portrait model. The detail enhancement LoRa module consists of the highest-quality subset of the portrait model's training data and only supports 1280 resolution training. The training data pairs are the candidate images selected in the above steps. High-quality samples are obtained by resizing the data to 1280×1280; low-quality samples are obtained by applying Gaussian blur to high-quality data, further resizing it to the (384, 512) range, and then enlarging it back to 1280×1280. The input prompt for the sample pairs is 'rich in detail' for high-quality samples and 'smooth in detail' for low-quality samples. The implementation uses high-quality and low-quality image sample pairs as input for differential training, thereby improving the skin texture of the generated portraits.

[0402] Figure 11 A schematic diagram illustrating the training process of the LoRa module for enhancing details. (e.g.) Figure 11 As shown, the model's input is a latent representation (xt, c, t), where xt is the image, c is the cue word, and ct is the original cue word. The high-quality image input is (xt, c+, t), where c+ is the original cue word concatenated with "rich in detail"; the low-quality image input is (xt, c-, t), where c- is the original cue word concatenated with "smooth in detail". The Unet module performs feature mapping based on the high-quality and low-quality image inputs, respectively, obtaining the corresponding outputs ∈θ(xt, c+, t) and ∈θ(xt, c-, t). During training, a loss function is used to constrain the difference between the two inputs, updating the weight parameters of the detail enhancement LoRa module.

[0403] In some embodiments of this application, the loss function described above is as shown in formula (2):

[0404]

[0405] It is understandable that the reason why detail enhancement LoRa is effective is that it controls variables so that only the single dimension of detail richness differs, while all other variables are completely consistent. In order to make the model focus only on the single dimension of detail quality difference, the loss function of the above formula (2) is designed as the model optimization objective.

[0406] in, This is the noise distribution predicted after optimizing the model parameters using low-quality samples. This is the noise distribution predicted after optimizing the model parameters using high-quality samples. Indicates a high-quality image, c + The prompt words that indicate high-quality images c represents a low-quality image. - The text represents the prompt word corresponding to the low-quality image, t is a specific time when noise is added, and ε is the true noise distribution to be fitted at time t. and Both are close to ε, but there is a significant difference in the level of detail between them. The lowest loss is achieved if and only if all other parts are completely identical except for this property, that is, the model only focuses on the changing parts.

[0407] For example, after training is complete, the weights of the detail enhancement LoRa are fused into the Unet of the initial model to obtain the portrait base model module in personalized beautification.

[0408] For example, combining the above examples and Figure 11 The initial model's visual training process is as follows: Figure 12 As shown, an image sample pair consisting of a high-quality image 61 and a low-quality image 62 is input into the initial model. The image detail cues for the low-quality image are "smooth skin details" and for the high-quality image are "rich skin details". The text encoder encodes the cues for both images into conditional features, and the VAE encoder converts the image pair into a latent representation. The conditional features and latent features are then input into UNet for differential feature mapping. Finally, the image is reconstructed by the VAE decoder, allowing the model to learn the visual difference between rich detail and smoothness, thus generating a high-quality image 63.

[0409] The following example illustrates the initial training process based on the sample image set. Exemplarily, the above process may include the following steps:

[0410] Step 41: For each candidate image, upsample the 1024x1024 candidate image to 1280x1280 to obtain high-quality image A. Apply Gaussian blur to high-quality image A, then downsample to 384x384, and then upsample back to 1280x1280 to obtain low-quality image B. Construct an image sample pair based on high-quality image A and low-quality image B.

[0411] Step 42: Construct the cue word "a woman, rich in detail" for high-quality image A; associate the cue word "a woman, smooth in detail" for low-quality image B.

[0412] Step 43: Input the constructed image sample pairs into the initial model, and output the predicted feature information of high-quality image A and low-quality image B.

[0413] Step 44: Based on the loss function, calculate the difference between the predicted feature information of high-quality image A and the predicted feature information of low-quality image B, and update the weights of the detail enhancement LoRa module based on the difference information, so that the model can learn the feature differences between the detail-rich image and the detail-smooth image.

[0414] In this embodiment, by enabling the model to learn the feature differences between "detail-rich" and "detail-smooth" images, the model can accurately generate images with corresponding details based on given text prompts during the inference stage. On the other hand, by performing quality differential training on the model, the model learns the ability to recover details from low-quality inputs, thereby enabling the model to generate images with richer details and closer to real skin texture.

[0415] The following, in conjunction with the accompanying drawings, provides an exemplary description of the workflow of a digital clone model, including a basic portrait model, a user Lora, and a beautification module.

[0416] For example, Figure 13 A flowchart illustrating the construction process of a digital avatar model, as shown below. Figure 13 As shown, the construction process of the digital avatar model is mainly divided into three stages: the first stage is the construction of the basic portrait model, the second stage is the user LoRa training, and the third stage is the training of the beautification module. Among them, steps 21 to 26 in the first stage are the training process of the basic portrait model.

[0417] For example, the process of building a basic human portrait model includes the following steps:

[0418] Step 21: Determine the sample image set based on the initial sample image set.

[0419] Step 22: Perform key point detection on each candidate sample image in the sample image set to identify the key regions in each candidate sample image.

[0420] Step 23: Determine the sample image set from all candidate sample images that meet the first condition.

[0421] Step 24: Scale and crop each sample image in the sample image set to generate a first-size image, a second-size image, and a third-size image corresponding to each sample image.

[0422] Step 25: Determine all first-size images as the first sample image set, all second-size images as the second sample image set, and all third-size images as the third sample image set. Train the initial model based on the first, second, and third sample image sets.

[0423] Step 26: Obtain at least one candidate image from the sample image set whose face region size is greater than the second size and whose face region proportion is greater than the second threshold, and obtain the complete face image and the partial face image corresponding to each candidate image.

[0424] Step 27: Train the initial model based on the complete face image and partial face image corresponding to each candidate image.

[0425] Step 28: The model training device constructs at least one set of face images based on the complete face image and the partial face image corresponding to each candidate image.

[0426] Step 29: Input the initial model based on at least one set of face images, train the initial model, and obtain the basic human image model.

[0427] Step 30: Select images by clustering them in the photo album.

[0428] Step 31: Train user Lora.

[0429] Step 32: Train the beauty module.

[0430] For example, the model training device first trains a basic portrait model, and then trains the user Lora and the beautification module based on the trained basic portrait model.

[0431] For example, the trained user Lora and the beautification module are integrated into the portrait base model to obtain the final customized portrait generation model. After the image is generated by this customized portrait generation model, the generated image is optimized through post-processing.

[0432] The above describes the training process for the basic portrait model. After the basic portrait model is trained, the user LoRa and the beautification module load the model parameters of the basic portrait model and train their respective modules based on their tasks. Finally, the two modules are superimposed on the Unet module of the basic portrait model to generate the user's portrait. The generated user portrait is then processed by the post-processing module and output to the user.

[0433] Figure 13 The entire flowchart clearly outlines the entire process from basic model preparation and personalized fine-tuning to final application.

[0434] This application provides an image generation method. Figure 14This is a schematic flowchart of the image generation method provided in the embodiments of this application, such as... Figure 14 As shown, the image generation method may include the following steps 301 and 302:

[0435] Step 301: The image generation device receives the original image and prompt words input by the user.

[0436] In some embodiments of this application, the original image described above may be an image provided by the user, such as one or more images selected by the user from a photo album.

[0437] In some embodiments of this application, the aforementioned prompts are text descriptions entered by the user, used to specify the desired image style, detail, or modification intent. For example, the user may enter prompts such as "a beautiful woman, rich in detail, with delicate skin" or "artistic oil painting style."

[0438] Step 302: The image generation device associates the original image and the prompt words with the input image generation model, performs model inference, and outputs the model-generated image.

[0439] In some embodiments of this application, the image generation device aligns and merges the original image with the prompt words, and constructs or edits the image based on the merged information, such as redrawing or detail enhancement, and outputs a new image that meets the user's needs, i.e., a model-generated image.

[0440] The image generation method provided in this application combines the original image with text prompts as joint input, allowing users to flexibly specify the desired effect of the generated image through text, which greatly improves the accuracy of the generated image; and the image generation model relies on high-quality sample images that have been selected for training, so it can output images with rich skin details and clear textures.

[0441] In some embodiments of this application, the image generation model described above includes a VAE encoder, a text encoder, a Unet module, and a VAE decoder; exemplarily, step 302 may include steps 302a to 302d:

[0442] Step 302a: The image generation device associates the original image and the prompt words with the input image generation model, and encodes the original image into image feature information through the VAE encoder.

[0443] Step 302b: The image generation device encodes the prompt words into text feature information using a text encoder.

[0444] Step 302c: The image generation device associates the image feature information and text feature information and inputs them into the Unet module for feature mapping processing, and outputs the predicted feature information.

[0445] Step 302d: The image generation device decodes the predicted feature information through the VAE decoder and outputs the model-generated image.

[0446] In some embodiments of this application, the image feature information is a vector representation of the original image in the latent space.

[0447] In some embodiments of this application, the text feature information is a feature vector obtained after the prompt words are encoded, which contains semantic information of the user instruction and is used to guide the direction of image generation.

[0448] In some embodiments of this application, the image generation apparatus can fuse image feature information and text feature information within Unet to form joint conditions.

[0449] In some embodiments of this application, the VAE decoder is used to decode feature vectors in the latent space and reconstruct them back into the pixel space to generate a visualization image.

[0450] It should be noted that VAE decoder is the same as VAE Decoder.

[0451] The image generation method provided in the embodiments of this application is illustrated by example below.

[0452] For example, in a specific application scenario, a user might want to optimize the skin texture in a photo. For instance... Figure 15 As shown, firstly, the user inputs a photo of themselves as the input image and enters the prompt "a beautiful woman with delicate skin and rich details" in the text input box; the image generation device associates the photo with the prompt and inputs it into the portrait base model trained based on the initial model; then, the VAE encoder 13a of the portrait generation model converts the photo into a latent representation, and the text encoder 13b encodes the prompt into conditional features, and the Unet module 13c performs model inference to generate predicted features; finally, the VAE decoder 13d reconstructs a beautified photo of the person with delicate skin and rich details based on the predicted features and outputs the photo.

[0453] In this embodiment, the image generation device maps the image and text to the same space through a VAE encoder and a text encoder, respectively, and then uses Unet to fuse the image and text to generate an image that meets the user's intent. Since the image generation model relies on high-quality sample images that have been selected for training, it can output high-quality images with rich skin details, clear textures, and that meet the user's intent.

[0454] It should be noted that the explanation of the method embodiments can be found in the relevant descriptions of the above embodiments, and will not be repeated here to avoid repetition.

[0455] The above-described method embodiments, or various possible implementations of the method embodiments, can be executed individually, or, provided there are no contradictions, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.

[0456] The model training method provided in this application can be executed by a model training device. This application uses an example of a model training device executing the model training method to illustrate the model training device provided in this application.

[0457] Figure 16 This is a schematic diagram of the structure of the model training device 600 provided in the embodiments of this application, as shown below. Figure 16 As shown, the model training device 600 includes a processing module 601, which is used to determine a sample image set based on an initial sample image set. The sample image set includes at least one sample image. Each sample image includes a key region, which includes a face region and a shoulder region. The width and height of the face region are both greater than or equal to a first threshold. The face region has pore features and no blemish features. The processing module is also used to train the initial model based on the sample image set to obtain an image generation model.

[0458] In some embodiments of this application, the processing module is further configured to obtain an initial sample image set before determining the sample image set based on the initial sample image set, the initial sample image set including at least one candidate sample image; the processing module is specifically configured to:

[0459] Keypoint detection is performed on each candidate sample image to identify key regions in each candidate sample image;

[0460] Based on the width and height of the key region in each candidate sample image, the key region is expanded outward, and the expanded key region is determined as the cropping region.

[0461] The cropped region of each candidate sample image is extracted to obtain the cropped sample image set;

[0462] The cropped sample image set is input into the multimodal scoring model, and the image quality information of each candidate sample image is output. The image quality information indicates whether there are pore features and blemish features in the face region of each candidate sample image.

[0463] The sample image set is determined based on the image quality information of each candidate sample image.

[0464] In some embodiments of this application, the above-described processing module is specifically used to determine a sample image set from all candidate sample images that satisfy the first condition;

[0465] The first condition includes: the image quality information indicates that the face region of the candidate sample image has pore features and no blemish features, and the width and height of the face region of the candidate sample image are both greater than or equal to the first threshold.

[0466] In some embodiments of this application, the above-mentioned processing module is further configured to scale and crop each sample image in the sample image set before training the initial model based on the sample image set, generating a first-size image, a second-size image, and a third-size image corresponding to each sample image, wherein the image size of the first-size image is smaller than the image size of the second-size image, and the image size of the second-size image is smaller than the image size of the third-size image; the processing module is further configured to determine all first-size images as the first sample image set, all second-size images as the second sample image set, and all third-size images as the third sample image set; the processing module is specifically configured to train the initial model based on the first sample image set, the second sample image set, and the third sample image set.

[0467] In some embodiments of this application, the above-described processing module is specifically used for:

[0468] All second-size images in the second sample image set are input into the initial model. The Unet module of the initial model is used to perform feature mapping processing on each second-size image to obtain the first predicted feature information of each second-size image.

[0469] The model parameters of the Unet module are updated based on the first difference information between the first predicted feature information of each second-size image and the reference feature information of each second-size image.

[0470] The reference feature information for each second-size image is obtained by performing latent space feature mapping on each second-size image through the VAE encoder of the image generation module.

[0471] In some embodiments of this application, the above-mentioned processing module is further configured to, after updating the model parameters of the Unet module based on the first difference information between the first predicted feature information of each second-size image and the reference feature information of each second-size image, input at least one third-size image from the third sample image set into the image generation model, and perform feature mapping processing on each third-size image through the Unet module to obtain the second predicted feature information of each third-size image; the processing module is further configured to update the model parameters of the Unet module based on the second difference information between the second predicted feature information and the reference feature information of the third-size image; wherein, the reference feature information of each third-size image is obtained by performing latent space feature mapping on each third-size image through the VAE encoder of the image generation module.

[0472] In some embodiments of this application, the above-described processing module is further configured to:

[0473] After updating the model parameters of the Unet module based on the second difference information, a reference Unet module is constructed based on the updated model parameters of the Unet module.

[0474] Input the reference feature information of the third-size image corresponding to each sample image into the reference Unet module, and output the third predicted feature information;

[0475] The reference feature information of the first-size image and the reference feature information of the third-size image corresponding to each sample image are fused to output the fused feature information.

[0476] The fused feature information is input into the Unet module, which outputs the fourth predicted feature information.

[0477] The weight parameters of the Unet module are updated based on the difference between the third and fourth predicted feature information.

[0478] In some embodiments of this application, the above-described processing module is further configured to:

[0479] After updating the weight parameters of the Unet module based on the difference information between the third and fourth predicted feature information, at least one first image sample pair is constructed based on the first and second size images corresponding to each sample image, and at least one second image sample pair is constructed based on the second and third size images corresponding to each sample image.

[0480] Input at least one first image sample pair into the initial model, and perform feature mapping processing on the at least one first image sample pair through the Unet module to output the fifth predicted feature information;

[0481] Input at least one second image sample pair into the initial model, and perform feature mapping processing on the at least one second image sample pair through the reference Unet module to output the sixth predicted feature information;

[0482] The model parameters of the Unet module are updated based on the difference between the fifth and sixth predicted feature information.

[0483] In some embodiments of this application, the above-described processing module is further configured to:

[0484] Before training the initial model based on the sample image set, at least one candidate image is obtained from the sample image set whose face region size is greater than the second size and whose face region proportion is greater than the second threshold.

[0485] Obtain the complete face image and partial face image corresponding to each candidate image;

[0486] The initial model is trained based on the sample image set, including:

[0487] The initial model is trained based on the complete face image and partial face image corresponding to each candidate image.

[0488] In some embodiments of this application, the above-described processing module is specifically used for:

[0489] Each candidate image is input into the face detection model, and the facial feature point information of each candidate image is output.

[0490] Based on the facial feature point information of each candidate image, the face is cropped from each candidate image to obtain the complete face image corresponding to each candidate image.

[0491] The complete face image corresponding to each candidate image is input into the fine segmentation model, and the binary mask image corresponding to each complete face image is output. The image pixel values ​​of the binary mask image include white area pixel values ​​and black area pixel values. The image region corresponding to the white area pixel values ​​is the face skin region, and the image region corresponding to the black area pixel values ​​is the non-face skin region.

[0492] The image regions in all complete face images that correspond to the mask image regions that satisfy the second condition are determined as the sampling regions; wherein, the second condition is that the image pixel values ​​of the mask image regions in the binary mask image are the white area pixel values;

[0493] According to the reference cropping size, the sampled area of ​​each complete face image is cropped, and the cropped image is determined as the local face image corresponding to each candidate image.

[0494] In some embodiments of this application, the above-described processing module is specifically used for:

[0495] Based on the complete face image and partial face image corresponding to each candidate image, construct at least one set of face images;

[0496] Input at least one set of face images into the initial model and output predicted face feature information;

[0497] The model parameters of the initial model are updated based on the differences between the predicted facial feature information and the feature information of all complete facial images.

[0498] In some embodiments of this application, the above-described processing module is specifically used for:

[0499] Each candidate image is resized to a first size to obtain a first quality image. The first quality image is then blurred, and the image size of the blurred first quality image is subsequently resized to a second size and then back to the first size to obtain a second quality image. The first size is larger than the second size.

[0500] Establish a correlation between a first quality image and a first image detail cue, and establish a correlation between a second quality image and a second image detail cue, and construct at least one image sample pair based on the first quality image and the second quality image, wherein the first image detail cue and the second image detail cue indicate image detail features;

[0501] At least one image sample pair is input into the initial model. The Unet module performs feature mapping processing on the at least one image sample pair to obtain the seventh predicted feature information of the first quality image and the eighth predicted feature information of the second quality image.

[0502] The model parameters of the initial model are updated based on the difference between the seventh and eighth predicted feature information.

[0503] The model training apparatus provided in this application determines a sample image set based on an initial sample image set. This sample image set includes at least one sample image; each sample image includes a key region, which includes a face region and a shoulder region. The width and height of the face region are both greater than or equal to a first threshold. The face region has pore features and no blemishes. The model training apparatus trains the initial model based on the sample image set to obtain an image generation model. This solution constructs a training set by selecting sample images containing face and shoulder regions, and whose skin exhibits both pore features and blemish-free characteristics. Training the initial model yields an image generation model, thereby enhancing the skin texture of the generated portrait. This improves the realism of skin texture while effectively avoiding blemish generation, thus improving the quality of the generated image.

[0504] The image generation method provided in this application can be executed by an image generation device. This application uses an image generation device executing a model training method as an example to illustrate the image generation device provided in this application.

[0505] Figure 17 This is a schematic diagram of the structure of the image generation apparatus 700 provided in the embodiments of this application, as shown below. Figure 17 As shown, the image generation device 700 includes an acquisition module 701 and a processing module 702; wherein, the acquisition module 701 is used to receive the original image and prompt words input by the user; the processing module 702 is used to associate the original image and prompt words received by the acquisition module with the image generation model of any one of claims 1-12 for model inference, and output the model-generated image.

[0506] In some embodiments of this application, the image generation model includes a VAE encoder, a text encoder, a Unet module, and a VAE decoder; the processing module is specifically used for:

[0507] The original image and the prompt words are associated with the input image generation model. The original image is then encoded into image feature information through a VAE encoder.

[0508] The prompt words are encoded into text feature information using a text encoder;

[0509] Image and text feature information are correlated and input into the Unet module for feature mapping processing, and predicted feature information is output.

[0510] The VAE decoder decodes the predicted feature information and outputs the model-generated image.

[0511] The image generation apparatus provided in this application combines the original image with text prompts as joint input, allowing users to flexibly specify the desired effect of the generated image through text, which greatly improves the accuracy of the generated image; and the image generation model relies on high-quality sample images that have been selected for training, thus it can output images with rich skin details and clear textures.

[0512] The model training device or image generation device in this application embodiment can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. The embodiments of this application do not specifically limit the scope.

[0513] The model training device or image generation device in the embodiments of this application can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application does not specifically limit the specific operating system used.

[0514] The model training apparatus provided in this application embodiment can implement the various processes implemented in the method embodiment of the model training method, and the image generation apparatus provided in this application embodiment can implement the various processes implemented in the method embodiment of the image generation method. To avoid repetition, these will not be described again here.

[0515] Optionally, such as Figure 18 As shown, this application embodiment also provides an electronic device 800, including a processor 801 and a memory 802. The memory 802 stores a program or instructions that can run on the processor 801. When the program or instructions are executed by the processor 801, they implement the various steps of the above-described model training method or image generation method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0516] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0517] Figure 19 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0518] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0519] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 19 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0520] In some possible implementations of this application, the processor 110 is used to determine a sample image set based on an initial sample image set, the sample image set including at least one sample image; wherein each sample image includes a key region, the key region including a face region and a shoulder region, the width and height of the face region are both greater than or equal to a first threshold, the face region has pore features and the face region does not have blemish features; the processor is also used to train an initial model based on the sample image set to obtain an image generation model.

[0521] In some embodiments of this application, the processor is further configured to obtain an initial sample image set, which includes at least one candidate sample image, before determining the sample image set based on the initial sample image set; the processor is specifically configured to:

[0522] Keypoint detection is performed on each candidate sample image to identify key regions in each candidate sample image;

[0523] Based on the width and height of the key region in each candidate sample image, the key region is expanded outward, and the expanded key region is determined as the cropping region.

[0524] The cropped region of each candidate sample image is extracted to obtain the cropped sample image set;

[0525] The cropped sample image set is input into the multimodal scoring model, and the image quality information of each candidate sample image is output. The image quality information indicates whether there are pore features and blemish features in the face region of each candidate sample image.

[0526] The sample image set is determined based on the image quality information of each candidate sample image.

[0527] In some embodiments of this application, the processor described above is specifically used to determine a sample image set from all candidate sample images that satisfy the first condition;

[0528] The first condition includes: the image quality information indicates that the face region of the candidate sample image has pore features and no blemish features, and the width and height of the face region of the candidate sample image are both greater than or equal to the first threshold.

[0529] In some embodiments of this application, the processor is further configured to scale and crop each sample image in the sample image set before training the initial model based on the sample image set, generating a first-size image, a second-size image, and a third-size image corresponding to each sample image, wherein the image size of the first-size image is smaller than the image size of the second-size image, and the image size of the second-size image is smaller than the image size of the third-size image; the processor is further configured to determine all first-size images as the first sample image set, all second-size images as the second sample image set, and all third-size images as the third sample image set; specifically, the processor is configured to train the initial model based on the first sample image set, the second sample image set, and the third sample image set.

[0530] In some embodiments of this application, the processor described above is specifically used for:

[0531] All second-size images in the second sample image set are input into the initial model. The Unet module of the initial model is used to perform feature mapping processing on each second-size image to obtain the first predicted feature information of each second-size image.

[0532] The model parameters of the Unet module are updated based on the first difference information between the first predicted feature information of each second-size image and the reference feature information of each second-size image.

[0533] The reference feature information for each second-size image is obtained by performing latent space feature mapping on each second-size image through the VAE encoder of the image generation module.

[0534] In some embodiments of this application, the processor is further configured to, after updating the model parameters of the Unet module based on the first difference information between the first predicted feature information of each second-size image and the reference feature information of each second-size image, input at least one third-size image from the third sample image set into the image generation model, and perform feature mapping processing on each third-size image through the Unet module to obtain the second predicted feature information of each third-size image; the processor is further configured to update the model parameters of the Unet module based on the second difference information between the second predicted feature information and the reference feature information of the third-size image; wherein, the reference feature information of each third-size image is obtained by performing latent space feature mapping on each third-size image through the VAE encoder of the image generation module.

[0535] In some embodiments of this application, the processor described above is further used for:

[0536] After updating the model parameters of the Unet module based on the second difference information, a reference Unet module is constructed based on the updated model parameters of the Unet module.

[0537] Input the reference feature information of the third-size image corresponding to each sample image into the reference Unet module, and output the third predicted feature information;

[0538] The reference feature information of the first-size image and the reference feature information of the third-size image corresponding to each sample image are fused to output the fused feature information.

[0539] The fused feature information is input into the Unet module, which outputs the fourth predicted feature information.

[0540] The weight parameters of the Unet module are updated based on the difference between the third and fourth predicted feature information.

[0541] In some embodiments of this application, the processor described above is further used for:

[0542] After updating the weight parameters of the Unet module based on the difference information between the third and fourth predicted feature information, at least one first image sample pair is constructed based on the first and second size images corresponding to each sample image, and at least one second image sample pair is constructed based on the second and third size images corresponding to each sample image.

[0543] Input at least one first image sample pair into the initial model, and perform feature mapping processing on the at least one first image sample pair through the Unet module to output the fifth predicted feature information;

[0544] Input at least one second image sample pair into the initial model, and perform feature mapping processing on the at least one second image sample pair through the reference Unet module to output the sixth predicted feature information;

[0545] The model parameters of the Unet module are updated based on the difference between the fifth and sixth predicted feature information.

[0546] In some embodiments of this application, the processor described above is further used for:

[0547] Before training the initial model based on the sample image set, at least one candidate image is obtained from the sample image set whose face region size is greater than the second size and whose face region proportion is greater than the second threshold.

[0548] Obtain the complete face image and partial face image corresponding to each candidate image;

[0549] The initial model is trained based on the sample image set, including:

[0550] The initial model is trained based on the complete face image and partial face image corresponding to each candidate image.

[0551] In some embodiments of this application, the processor described above is specifically used for:

[0552] Each candidate image is input into the face detection model, and the facial feature point information of each candidate image is output.

[0553] Based on the facial feature point information of each candidate image, the face is cropped from each candidate image to obtain the complete face image corresponding to each candidate image.

[0554] The complete face image corresponding to each candidate image is input into the fine segmentation model, and the binary mask image corresponding to each complete face image is output. The image pixel values ​​of the binary mask image include white area pixel values ​​and black area pixel values. The image region corresponding to the white area pixel values ​​is the face skin region, and the image region corresponding to the black area pixel values ​​is the non-face skin region.

[0555] The image regions in all complete face images that correspond to the mask image regions that satisfy the second condition are determined as the sampling regions; wherein, the second condition is that the image pixel values ​​of the mask image regions in the binary mask image are the white area pixel values;

[0556] According to the reference cropping size, the sampled area of ​​each complete face image is cropped, and the cropped image is determined as the local face image corresponding to each candidate image.

[0557] In some embodiments of this application, the processor described above is specifically used for:

[0558] Based on the complete face image and partial face image corresponding to each candidate image, construct at least one set of face images;

[0559] Input at least one set of face images into the initial model and output predicted face feature information;

[0560] The model parameters of the initial model are updated based on the differences between the predicted facial feature information and the feature information of all complete facial images.

[0561] In some embodiments of this application, the processor described above is specifically used for:

[0562] Each candidate image is resized to a first size to obtain a first quality image. The first quality image is then blurred, and the image size of the blurred first quality image is subsequently resized to a second size and then back to the first size to obtain a second quality image. The first size is larger than the second size.

[0563] Establish a correlation between a first quality image and a first image detail cue, and establish a correlation between a second quality image and a second image detail cue, and construct at least one image sample pair based on the first quality image and the second quality image, wherein the first image detail cue and the second image detail cue indicate image detail features;

[0564] At least one image sample pair is input into the initial model. The Unet module performs feature mapping processing on the at least one image sample pair to obtain the seventh predicted feature information of the first quality image and the eighth predicted feature information of the second quality image.

[0565] The model parameters of the initial model are updated based on the difference between the seventh and eighth predicted feature information.

[0566] The electronic device provided in this application embodiment determines a sample image set based on an initial sample image set. This sample image set includes at least one sample image. Each sample image includes a key region, which includes a face region and a shoulder region. The width and height of the face region are both greater than or equal to a first threshold. The face region has pore features and no blemishes. A model training device trains an initial model based on the sample image set to obtain an image generation model. This solution constructs a training set by selecting sample images containing face and shoulder regions, and whose skin exhibits both pore features and blemish-free characteristics. The initial model is then trained to obtain an image generation model, thereby enhancing the skin texture of the generated portrait. This improves the realism of the skin texture while effectively avoiding blemish generation, thus improving the quality of the generated image.

[0567] In some possible implementations of this application, the processor 110 is used to receive the original image and prompt words input by the user; the processor 110 is also used to associate the original image and prompt words received by the acquisition module with the input image generation model to perform model inference and output the model-generated image.

[0568] In some embodiments of this application, the image generation model includes a VAE encoder, a text encoder, a Unet module, and a VAE decoder; the processor is specifically used for:

[0569] The original image and the prompt words are associated with the input image generation model. The original image is then encoded into image feature information through a VAE encoder.

[0570] The prompt words are encoded into text feature information using a text encoder;

[0571] Image and text feature information are correlated and input into the Unet module for feature mapping processing, and predicted feature information is output.

[0572] The VAE decoder decodes the predicted feature information and outputs the model-generated image.

[0573] The electronic device provided in this application combines the original image with text prompts as joint input, allowing users to flexibly specify the desired effect of the generated image through text, which greatly improves the accuracy of the generated image; and the image generation model relies on high-quality sample images that have been selected for training, so it can output images with rich skin details and clear textures.

[0574] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0575] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0576] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0577] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described model training method or the various processes of the above-described image generation method, and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0578] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0579] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described model training method or the various processes of the above-described image generation method, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0580] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0581] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the method embodiment of the model training method described above or the various processes of the method embodiment of the image generation method described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0582] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0583] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0584] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A model training method, characterized in that the method comprises: determining a sample image set based on an initial sample image set, the sample image set comprising at least one sample image; wherein each sample image comprises a key region, the key region comprising a face region and a shoulder region, the face region having a width and a height both greater than or equal to a first threshold, the face region having a pore feature and the face region not having a flaw feature; training an initial model based on the sample image set to obtain an image generation model.

2. The method of claim 1, wherein, Before the determining of the sample image set based on the initial sample image set, the method further comprises: obtaining an initial sample image set, the initial sample image set comprising at least one candidate sample image; the determining of the sample image set based on the initial sample image set comprises: performing key point detection on each candidate sample image to identify a key region in each candidate sample image; extending the key region in each candidate sample image outward based on a width and a height of the key region, and determining an extended key region as a cropped region; cropping an image of the cropped region of each candidate sample image to obtain a cropped sample image set; inputting the cropped sample image set into a multi-modal scoring model to output image quality information of each candidate sample image, the image quality information indicating whether the face region in each candidate sample image has a pore feature and whether the face region has a flaw feature; determining the sample image set based on the image quality information of each candidate sample image.

3. The method of claim 2, wherein, The determining of the sample image set based on the image quality information of each candidate sample image comprises: determining all candidate sample images satisfying a first condition as the sample image set; wherein the first condition comprises that the image quality information indicates that the face region of the candidate sample image has a pore feature and the face region does not have a flaw feature, and the width and the height of the face region of the candidate sample image are both greater than or equal to the first threshold.

4. The method according to any one of claims 1 to 3, characterized in that, Before the training of the initial model based on the sample image set, the method further comprises: performing scaling and cropping processing on each sample image in the sample image set to generate a first size image, a second size image and a third size image corresponding to each sample image, the image size of the first size image being smaller than the image size of the second size image, and the image size of the second size image being smaller than the image size of the third size image; determining all first size images as a first sample image set, determining all second size images as a second sample image set, and determining all third size images as a third sample image set; the training of the initial model based on the sample image set comprises: training the image generation model based on the first sample image set, the second sample image set and the third sample image set.

5. The method of claim 4, wherein, the training of the initial model based on the sample image set comprises: inputting all second-size images in the second sample image set into the initial model, performing feature mapping processing on each second-size image by a Unet module of the initial model to obtain first predicted feature information of each second-size image; updating model parameters of the Unet module based on first difference information between the first predicted feature information of each second-size image and reference feature information of each second-size image; wherein the reference feature information of each second-size image is obtained by performing latent space feature mapping on each second-size image by a VAE encoder of the image generation module.

6. The method of claim 5, wherein, After the model parameters of the Unet module are updated based on the first difference information between the first predicted feature information of each second-size image and the reference feature information of each second-size image, the method further comprises: inputting at least one third-size image in the third sample image set into the image generation model, performing feature mapping processing on each third-size image by the Unet module to obtain second predicted feature information of each third-size image; updating the model parameters of the Unet module based on second difference information between the second predicted feature information and reference feature information of the third-size image; wherein the reference feature information of each third-size image is obtained by performing latent space feature mapping on each third-size image by the VAE encoder of the image generation module.

7. The method of claim 6, wherein, After the model parameters of the Unet module are updated based on the second difference information, the method further comprises: constructing a reference Unet module based on the updated model parameters of the Unet module; inputting the reference feature information of the third-size image corresponding to each sample image into the reference Unet module to output third predicted feature information; performing feature fusion on the reference feature information of the first-size image and the reference feature information of the third-size image corresponding to each sample image to output fusion feature information; inputting the fusion feature information into the Unet module to output fourth predicted feature information; updating weight parameters of the Unet module based on difference information between the third predicted feature information and the fourth predicted feature information.

8. The method of claim 7, wherein, After the weight parameters of the Unet module are updated based on the difference information between the third predicted feature information and the fourth predicted feature information, the method further comprises: constructing at least one first image sample pair based on the first-size image and the second-size image corresponding to each sample image, and constructing at least one second image sample pair based on the second-size image and the third-size image corresponding to each sample image; inputting the at least one first image sample pair into the initial model, performing feature mapping processing on the at least one first image sample pair by the Unet module to output fifth predicted feature information; inputting the at least one second image sample pair into the initial model, performing feature mapping processing on the at least one second image sample pair by the reference Unet module to output sixth predicted feature information; Update the model parameters of the Unet module based on difference information between the fifth predicted feature information and the sixth predicted feature information.

9. The method according to any one of claims 1 to 3, characterized in that, Before the training of the initial model based on the sample image set, the method further comprises: Obtaining at least one candidate image in the sample image set, whose face region size is greater than a second size and whose face region proportion is greater than a second threshold value; Obtaining a complete face image and a local face image corresponding to each candidate image; The training of the initial model based on the sample image set comprises: Training the initial model based on the complete face image and the local face image corresponding to each candidate image.

10. The method of claim 9, wherein, The obtaining of the complete face image and the local face image corresponding to each candidate image comprises: Inputting each candidate image into a face detection model to output face feature point information of each candidate image; Performing face cropping on each candidate image based on the face feature point information of each candidate image to obtain a complete face image corresponding to each candidate image; Inputting the complete face image corresponding to each candidate image into a fine segmentation model to output a binary mask image corresponding to each complete face image, wherein the image pixel value of the binary mask image comprises a white region pixel value and a black region pixel value, the image region corresponding to the white region pixel value is a face skin region, and the image region corresponding to the black region pixel value is a non-face skin region; Determining an image region corresponding to a mask image region satisfying a second condition in all complete face images as a sampling region, wherein the second condition is that the image pixel value of the mask image region in the binary mask image is a white region pixel value; According to a reference cropping size, cropping the image of the sampling region of each complete face image to determine the cropped image as a local face image corresponding to each candidate image.

11. The method according to claim 9 or 10, characterized in that, The training of the initial model based on the complete face image and the local face image corresponding to each candidate image comprises: Based on the complete face image and the local face image corresponding to each candidate image, at least one group of face images is constructed. Inputting the at least one group of face images into the initial model to output predicted face feature information; Based on difference information between the predicted face feature information and the feature information of all complete face images, updating the model parameters of the initial model.

12. The method of claim 9 or 10, wherein, The training of the initial model based on the sample image set comprises: Adjusting the image size of each candidate image to a first size to obtain a first quality image, performing blur processing on the first quality image, and sequentially adjusting the image size of the first quality image after the blur processing to a second size and the first size to obtain a second quality image; the first size is greater than the second size; Establishing an association relationship between the first quality image and a first image detail prompt word, and an association relationship between the second quality image and a second image detail prompt word, and constructing at least one image sample pair based on the first quality image and the second quality image, wherein the first image detail prompt word and the second image prompt word indicate image detail features; input the at least one image sample pair into the initial model, perform feature mapping processing on the at least one image sample pair through the Unet module, and obtain seventh predicted feature information of the first quality image and eighth predicted feature information of the second quality image; update the model parameters of the initial model based on difference information between the seventh predicted feature information and the eighth predicted feature information.

13. An image generation method characterized by, The method comprises: receiving a user-input original image and a prompt word; inputting the original image and the prompt word into the image generation model of any one of claims 1-12 for model inference, and outputting a model-generated image.

14. The method of claim 13, wherein, The image generation model comprises a VAE encoder, a text encoder, a Unet module, and a VAE decoder. The image generation model comprises a VAE encoder, a text encoder, a Unet module, and a VAE decoder. The image generation model comprises a VAE encoder, a text encoder, a Unet module, and a VAE decoder. inputting the original image and the prompt word into the image generation model, encoding the original image into image feature information through the VAE encoder; encoding the prompt word into text feature information through the text encoder; inputting the image feature information and the text feature information into the Unet module for feature mapping processing, and outputting predicted feature information; decoding the predicted feature information through the VAE decoder, and outputting a model-generated image.

15. A model training apparatus, characterized in that the apparatus comprises:

16. The apparatus of claim 15, wherein, a processing module configured to determine a sample image set based on an initial sample image set, the sample image set comprising at least one sample image, wherein each sample image comprises a key region, the key region comprising a face region and a shoulder region, the face region having a width and a height both greater than or equal to a first threshold, the face region having a pore feature and the face region being free of a flaw feature, and train an initial model based on the sample image set to obtain an image generation model. The processing module is further configured to obtain an initial sample image set before determining the sample image set based on the initial sample image set, the initial sample image set comprising at least one candidate sample image. The processing module is specifically configured to: perform key point detection on each candidate sample image to identify a key region in each candidate sample image; perform outer expansion on the key region in each candidate sample image based on the width and the height of the key region, and determine an expanded key region as a cropped region; obtain a cropped sample image set by cropping the image of the cropped region of each candidate sample image; input the cropped sample image set into a multi-modal scoring model to output image quality information of each candidate sample image, the image quality information indicating whether the face region in each candidate sample image has a pore feature and whether the face region has a flaw feature; 17. The apparatus of claim 16, wherein, determine the sample image set based on the image quality information of each candidate sample image. The processing module is specifically configured to determine all candidate sample images satisfying a first condition as the sample image set. The first condition comprises: the image quality information indicates that the face region of the candidate sample image has a pore feature and does not have a flaw feature, and the width and height of the face region of the candidate sample image are greater than or equal to a first threshold.

18. The apparatus of any one of claims 15 to 17, wherein, The processing module is further configured to perform scaling and cutting processing on each sample image in the sample image set to generate a first size image, a second size image and a third size image corresponding to each sample image before training an initial model based on the sample image set, wherein the image size of the first size image is smaller than the image size of the second size image, and the image size of the second size image is smaller than the image size of the third size image. The processing module is further configured to determine all the first size images as a first sample image set, determine all the second size images as a second sample image set, and determine all the third size images as a third sample image set. The processing module is specifically configured to train the image generation model based on the first sample image set, the second sample image set and the third sample image set.

19. An electronic device, comprising: A device comprising a processor and a memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implement the model training method according to any one of claims 1-12.

20. An electronic device, comprising: A device comprising a processor and a memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implement the steps of the image generation method according to claim 13 or 14.

Images