Intelligent generation method and device for virtual model clothing display images based on diffusion model

Abstract

The present invention discloses a method and device for intelligently generating virtual model clothing display images based on a diffusion model. The original data in the data set is processed with a clothing segmentation model to obtain a target clothing image and a posture image, and a clothing and posture reconstruction variational autoencoder and an adaptive posture generation model M1 are trained; the original data in the data set is spatially spliced ​​to obtain combined data; the combined data is used to fine-tune the conditional diffusion model, and clothing enhancement is performed from feature fusion and guided generation to obtain a clothing enhanced display image generation model M2; an image of a base mannequin or an arbitrary person wearing target clothing is input, and a preliminary clothing display image is obtained through M1 and M2, and the final clothing display image is optimized using a face repair model. The present invention realizes innovation in terms of training mode, clothing feature retention and generation results, and can realize convenient operation and high-quality generation of virtual model clothing display images.

Description

Technical Field

[0001] The present invention relates to the field of image generation, and in particular to a method and a device for intelligently generating virtual model clothing display images based on a diffusion model. Background Art

[0002] With the development of artificial intelligence (AI), generative artificial intelligence (AIGC) technology has been able to automatically generate high-quality digital content such as images and videos. Therefore, AIGC technology has also been widely used in clothing display tasks to generate specific clothing display images. However, the current clothing display image generation technology has problems such as relying on paired data training, failing to maintain information such as clothing pattern and texture, and insufficient generation diversity. It cannot bring real wearing effects to consumers and is difficult to fully meet the needs of different merchants and consumers, which limits the wide application and further promotion of AIGC in clothing display tasks.

[0003] Most of the current clothing display image generation is based on real model clothing image generation. According to the technical context of the generative model, it can be divided into methods based on CNN or GAN. These methods approximate the deformation of clothing through geometric matching of two-dimensional images, but this kind of deformation is only obtained by mapping and twisting the image, and cannot simulate the subtle changes of clothing such as wrinkles and flutter in the real world. With the development of the Diffusion model, a method for generating real model clothing images based on the Diffusion model has gradually emerged. Although the Diffusion model has demonstrated its potential in generating high-quality images in clothing displays, when the target is complex clothing, there are still problems such as insufficient detail retention and low control accuracy. In terms of data, the diversity and detailed features of complex clothing also make it difficult to build a sufficiently comprehensive and high-quality dataset, which may lead to poor performance of the model when dealing with complex clothing that has not been seen before.

[0004] In summary, the current clothing display image generation technology is developing rapidly, and the quality of the generated results is getting higher and higher. However, most of the current clothing display methods are based on real model clothing display methods generated by existing real models. Such methods usually require paired data sets corresponding to clothing and models for training, and this method also has problems such as difficulty in retaining clothing texture details and single model features in the generated images. At the same time, because the clothing display images generated by such methods retain the existing models, merchants do not have copyrights and cannot be directly displayed commercially.

[0005] In view of the shortcomings of previous clothing display image generation methods, the present invention proposes an intelligent generation method for virtual model clothing display images based on a diffusion model. This method breaks through the limitations of paired data on model training, and while more effectively retaining clothing texture details, generates multi-pose virtual model clothing display images of matching clothing through prior target clothing. This method has more diverse generation results and allows merchants to directly conduct commercial clothing displays. Summary of the invention

[0006] In view of the above shortcomings, the present invention provides a method and device for intelligently generating virtual model clothing display images based on a diffusion model. The method only requires an image of a plain mannequin or any person wearing target clothing to generate virtual model clothing display images in various postures. The method innovatively proposes a new mode of clothing expansion. At the same time, it integrates clothing adaptive posture generation and classifier-free guidance modules to better retain clothing details, improve generation quality and diversity, and achieve higher quality virtual model clothing display image generation.

[0007] In order to achieve the above-mentioned object, the technical solution of the present invention is as follows: In a first aspect, the present invention provides a method for intelligently generating a virtual model clothing display image based on a diffusion model, the method comprising the following steps:

[0008] (1) Obtain a paired data set of {clothing image, model clothing display image} and an unpaired data set containing only model clothing display images, and segment the target clothing image of the model clothing display image; perform pose estimation and text description extraction at the same time to obtain a data pair of {text description, target clothing image, pose image};

[0009] (2) Based on the data in step (1), the conditional diffusion model is trained to obtain an adaptive posture generation model M1;

[0010] (3) Dividing the area except the clothing in the model clothing display image into the to-be-generated area, obtaining the to-be-generated image and the to-be-generated mask, splicing the clothing image with the to-be-generated image, the posture image and the model clothing display image in the spatial dimension, and splicing the to-be-generated mask with a mask of the same size initialized to 0;

[0011] (4) Based on the spliced ​​image and mask training conditional diffusion model, the clothing enhanced display image generation model M2 is obtained, which is used to enhance clothing details from the two parts of feature fusion and guided generation;

[0012] (5) Obtain an image that needs to be displayed on a virtual model, segment out a target clothing image, input it into model M1, obtain a generated posture image adapted to the clothing, input the target clothing image and the generated posture image into model M2 together, obtain a preliminary virtual model clothing display image, and obtain an optimized final virtual model clothing display image based on the face repair model.

[0013] Furthermore, the step (1) comprises the following steps:

[0014] (1.1) Obtain a paired data set containing {clothing image, model clothing display image} data pairs and an unpaired data set containing only clothing display images for data processing, scale the data set to a specified size, convert the data type, and normalize the data;

[0015] (1.2) Using the pre-trained clothing segmentation model on the model clothing display image, obtain the clothing segmentation mask of the original image. The size of the clothing segmentation mask is equal to the original image, and the value of the clothing part is 1, and the value of the rest is 0. The target clothing image is obtained according to the clothing segmentation mask;

[0016] (1.3) Perform posture estimation and text description extraction on the dataset to obtain the posture image corresponding to the model clothing display image and construct the target clothing image obtained through step (1.2) into data D1 of {text description, target clothing image, posture image}.

[0017] Furthermore, the step (2) comprises the following steps:

[0018] (2.1) The VAE reconstruction variational autoencoder is an encoder-decoder structure, which includes an encoder with 22 convolution layers, 10 residual blocks and a symmetrical decoder;

[0019] (2.2) VAE reconstruction variational autoencoder uses clothing images and posture images as training data, and uses them as input after fixed scaling, type conversion, and normalization;

[0020] (2.3)VAE reconstruction variational autoencoder loss consists of reconstruction loss and perceptual loss;

[0021] (2.4) Use the Adam optimizer for optimization and adjust the learning rate and batch size hyperparameters. After the parameters are set, start training. After the training is completed, the clothing image reconstruction variational autoencoder R1 and the posture reconstruction variational autoencoder R2 are obtained;

[0022] (2.5) The adaptive posture generation model M1 is based on U-Net as the basic network, including VAE encoder, U-Net network and VAE decoder;

[0023] (2.6) The adaptive posture generation model M1 uses the data D1 in step (1.3) as training data, encodes the target clothing image and posture image into the latent space, and uses them as the input of the U-Net network after the concat operation of the channel dimension;

[0024] (2.7) In the noise adding process of the conditional diffusion model, noise ∈ is added to the latent representation z step by step. After time step t, the latent variable after noise addition is z t , in the denoising process, train a denoising network ∈ θ , predict z t The noise added in step t∈ θ (z t ,t), z t Subtract the predicted noise to get the predicted z t-1 , iterate this process, and finally gradually restore the potential representation z t For the predicted initial potential representation z0, the loss function of the entire training process is as follows:

[0025]

[0026] in, represents the mean square error loss, z t represents the pose encoding after adding noise in step t, z c represents clothing code, ∈ represents noise, ∈ θ (z t ,t,z c ) is the noise prediction network, used to predict z0 to z t Noise; represents the square of the L2 norm, that is, the sum of the squares of each element;

[0027] (2.8) Use AdamW optimizer for optimization and adjust the learning rate and batch size hyperparameters. After the parameters are set, start training. After the training is completed, the adaptive posture generation model M1 is obtained.

[0028] Furthermore, the step (3) comprises the following steps:

[0029] (3.1) The clothing segmentation mask obtained in step (1.2) is used as the mask to be generated, the clothing part has a value of 1, and the rest is the part to be generated with a value of 0, and the target clothing image obtained in step (1.2) is used as the image to be generated;

[0030] (3.2) The mask to be generated obtained in step (3.1) and the image to be generated are used to construct corresponding data D2 of {text description, clothing image, mask to be generated, image to be generated, posture image, model clothing display image};

[0031] (3.3) Using the data D2 obtained in step (3.2), the clothing image is spliced ​​with the image to be generated, the posture image, and the model clothing display image in the spatial dimension to obtain a combined image to be generated, a combined posture image, and a combined model clothing display image with a width twice that of the original image;

[0032] (3.4) Using the mask to be generated obtained in step (3.1), initialize a mask of the same size as the mask to be generated and whose value is all 0, and concatenate the two in the spatial dimension to obtain a combined mask to be generated whose width is twice that of the original mask;

[0033] (3.5) The data obtained in step (3.3) and step (3.4) are constructed to obtain corresponding data D3 of {text, combined mask to be generated, combined image to be generated, combined posture image, combined model clothing display image}.

[0034] Furthermore, the step (4) comprises the following steps:

[0035] (4.1) The clothing enhanced display image generation model M2 is based on U-Net as the basic network, including VAE encoder, U-Net network, VAE decoder, and CILP text encoder;

[0036] (4.2) The model uses the data D3 in step (3.5) as training data, and uses feature fusion and guided generation to enhance clothing details;

[0037] (4.3) In the noise adding process of the conditional diffusion model, noise ∈ is added to the latent representation z step by step. After time step t, the latent variable after noise addition is z t , in the denoising process, train a denoising network ∈ θ , predict z t The noise added in step t∈ θ (z t ,t), z t Subtract the predicted noise to get the predicted z t-1 , iterate this process, and finally gradually restore the potential representation z t For the predicted initial potential representation z0, the loss function of the entire training process is as follows:

[0038]

[0039] in, represents the mean square error loss, z t represents the clothing display image encoding after adding noise in step t, z c represents clothing conditional encoding, m represents the downsampled mask to be generated, x represents the input model clothing display image, ⊙ represents element-by-element multiplication, z prepresents the pose conditional encoding, e represents the text feature of CLIP encoding, z m⊙x represents the conditional coding of the image to be generated, ∈ represents noise, ∈ θ (z t ,z m⊙x ,z c ,t,m,z p ,e) is the noise prediction network, used to predict z0 to z t Noise; represents the square of the L2 norm, that is, the sum of the squares of each element;

[0040] (4.4) Use AdamW optimizer for optimization and adjust the learning rate and batch size hyperparameters. After the parameters are set, start training. After the training is completed, the clothing enhanced display image generation model M2 is obtained.

[0041] Furthermore, the feature fusion method used in step (4.2) includes: using feature space connection, concatenating data D2 with the corresponding clothing image or mask in the spatial dimension to obtain data D3, and then sending it to the encoder to obtain each potential representation, and concatenating all the potential representations as the input of the U-Net network; using the attention mechanism, injecting the text features encoded by the large-scale text image pre-training model CLIP into the model through cross-attention to control the global features of the generated image, and taking clothing as an additional condition, combining it with other conditions in the spatial dimension, and establishing a direct connection between any two positions of the image through the self-attention mechanism in the denoising U-Net, capturing long-distance dependencies, optimizing the intersection area of ​​the target clothing and the human body part, and further merging the extracted clothing texture details with the character image;

[0042] The guided generation method used in step (4.2) includes: using an additional classifier-free guided CFG technology to generate conditional guidance of the image. CFG calculates the residual of the conditional generation and unconditional generation prediction results, and uses the conditional guidance coefficient to amplify the influence of the condition in the generation process, so as to achieve the effect of conditional control enhancement. Step (4.2) uses an improved CFG that uses joint text conditions and clothing conditions to improve the generation effect of the final image. The specific formula is as follows:

[0043] ∈ θ (z t ,c t ,c c )=∈ θ (z t )+S c *(∈ θ (z t ,c c )-∈ θ (zt ))+S t *(∈ θ (z,c t ,c c )

[0044] -∈ θ (z t ,c c ))

[0045] Among them, ∈ θ (z t ,c t ) represents the text condition c t The result of guiding prediction, ∈ θ (z t ) represents the result of the prediction without text condition, S t is the text conditional guidance coefficient, ∈ θ (z t ,c t ,c c ) represents the text condition c t With clothing conditions c c The prediction results of joint guidance, S c is the clothing condition guidance coefficient.

[0046] Furthermore, the step (5) comprises the following steps:

[0047] (5.1) For an image of a mannequin or any person wearing target clothing, use the pre-trained clothing segmentation model to segment the image and obtain the input target clothing;

[0048] (5.2) Input the target clothing into the adaptive posture model M1 to obtain the generated posture of the adaptive clothing;

[0049] (5.3) Inputting the target clothing and generated posture into the clothing enhanced display image generation model M2 to obtain a preliminary virtual model clothing display image;

[0050] (5.4) Using the pre-trained model to perform face detection on the obtained preliminary virtual model clothing display image, cropping the detected face in the image, and scaling it to a fixed size;

[0051] (5.5) For the obtained face, the high-quality face image blocks in the pre-trained codebook are used in combination with Transformer to capture global face information, realize blind face repair of low-quality or damaged facial images, and obtain the optimized final virtual model clothing display image.

[0052] In a second aspect, the present invention also provides a device for intelligently generating virtual model clothing display images based on a diffusion model, comprising a memory and one or more processors, wherein the memory stores executable code, and wherein when the processor executes the executable code, it implements the method for intelligently generating virtual model clothing display images based on a diffusion model.

[0053] In a third aspect, the present invention further provides a computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the method for intelligently generating virtual model clothing display images based on a diffusion model is implemented.

[0054] In a fourth aspect, the present invention further provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the method for intelligently generating virtual model clothing display images based on a diffusion model.

[0055] The beneficial effects of the present invention are as follows: only one image of a plain mannequin or any person wearing target clothing is needed to generate virtual model clothing display images with various postures; the present method innovatively proposes a new clothing expansion generation mode, and the model can construct a (clothing, model image) data pair by itself through a clothing segmentation model according to a single clothing display image. This method greatly increases the available data, so that the model does not need to rely on paired data for training and is not constrained by data scarcity; the present method innovatively combines feature fusion and guided enhancement to enhance clothing details, and maximizes the retention of clothing pattern, texture details and other features through feature space connection, attention mechanism and conditional guidance of joint text conditions and clothing conditions; the present method innovatively combines an adaptive posture generation module, and through additional posture information, the model generation result has a better human body structure, and the final virtual model has a variety of postures, and the virtual model clothing display image generation method of the present invention can generate virtual models that do not exist in the world, avoid copyright disputes, and e-commerce merchants can directly use the generated images for online clothing display. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 The present invention provides a flow chart of a method for intelligently generating virtual model clothing display images based on a diffusion model.

[0057] Figure 2 This is an example of the result of clothing segmentation.

[0058] Figure 3 This is an example of the results of the adaptive posture model.

[0059] Figure 4 This is an example of the results of the clothing enhanced display image generation model.

[0060] Figure 5 This is an example of the generated multi-pose virtual model clothing display image results.

[0061] Figure 6 It is a structural diagram of a device for intelligently generating virtual model clothing display images based on a diffusion model provided by the present invention. DETAILED DESCRIPTION

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, the present invention is further described below with reference to the accompanying drawings.

[0063] Aiming at the task of generating clothing display images for practical applications, the present invention proposes an intelligent generation method for virtual model clothing display images based on a diffusion model. The method can generate virtual model clothing display images with multiple postures through an image of a human body model or any person wearing target clothing. After clothing segmentation of a single input clothing display image, the target clothing is obtained, which can form a {clothing image, model clothing display image} data pair with the input clothing display image, which makes the method break through the shackles of traditional real model clothing display images that can only be trained using paired data. Considering that the human body structure generation effect is not good when only using the target clothing as input to directly generate clothing display images, an adaptive posture generation module is added. By taking the target clothing as input, multiple different postures adapted to clothing are generated, and the posture is used as additional information to guide the generation of the human body structure of the clothing display image, and the diversity of the clothing display image generation results is also improved. In order to further strengthen the retention of clothing pattern, texture and other detail information, the method also performs clothing enhancement from two parts: feature fusion and guided generation. Finally, a pre-trained face repair model is used to improve the quality of human body details in the final virtual model clothing display image.

[0064] Example 1

[0065] like Figure 1 As shown, an embodiment of the present invention provides a method for intelligently generating a virtual model clothing display image based on a diffusion model, and the specific steps are as follows:

[0066] (1) The paired dataset Dresscode containing data pairs of {clothing image, model clothing display image} and the unpaired dataset Deepfashion containing only clothing display images are preprocessed. The target clothing images of the model clothing display images are segmented using the pre-trained clothing segmentation model, and pose estimation and text description extraction are performed to construct the corresponding data D1 of {text description, target clothing image, pose image}.

[0067] (1.1) All data in the dataset are scaled to (512, 384) size, converted to tensor format and normalized to [-1, 1].

[0068] (1.2) Use the pre-trained clothing segmentation model huggingface-cloth-segmentation on the model clothing display image to obtain the clothing segmentation mask of the original image. The size of the clothing segmentation mask is equal to the original image, and the value of the clothing part is 1, and the value of the rest is 0. Multiply the clothing segmentation mask with the model clothing display image to obtain the target clothing image, as shown in Figure 2 shown.

[0069] (1.3) Use the pre-trained human pose estimation model and BLIP text description extraction model on the model clothing display images in the dataset to obtain the pose image and text description of the original image.

[0070] (1.4) The posture image, text description and target clothing image obtained by step (1.2) corresponding to the model clothing display image in the data set are constructed into {text description, target clothing image, posture image} data D1, and each pair of target clothing image and posture image corresponds to the same model clothing display image.

[0071] (2) Use the data D1 constructed in step (1) to train the VAE reconstruction variational autoencoder of the target clothing image and posture image, and obtain the clothing image reconstruction variational autoencoder R1 and the posture reconstruction variational autoencoder R2.

[0072] (2.1) The model structure of the VAE reconstruction variational autoencoder is an encoder-decoder structure, which includes an encoder with 22 convolution layers, 10 residual blocks and a symmetrical decoder.

[0073] (2.2) The model uses {target clothing image, posture image} as training data, scaled to (512, 384) size, transformed into a tensor, and normalized to [-1, 1].

[0074] (2.3) VAE reconstruction The variational autoencoder loss consists of reconstruction loss and perceptual loss. The reconstruction loss is L1loss. Specifically, for an input image I with N pixels, the reconstruction result is Z, then the L1 loss here can be expressed as:

[0075]

[0076] where |·| represents the absolute value, I i and Z i Represent the i-th pixel of the input image and the reconstructed result image respectively.

[0077] Perceptual loss using pre-trained deep convolutional neural networks Get some feature representations of the image, and use the Euclidean distance to measure the difference between these features of the original image and the reconstructed image. For the input image I and the reconstructed result Z, the perceptual loss here can be expressed as:

[0078]

[0079] Among them, j represents the number of layers of the network, C j H j W j Indicates the feature dimension size of the image at the jth layer of the network, Represents the square of the L2 norm, that is, the sum of the squares of each element.

[0080] The final loss function is defined as:

[0081]

[0082] (2.4) For the dataset used, it is divided into training data and test data with a ratio of 9:1, and the data is scaled to a size of (512, 384). The training is optimized using the Adam optimizer, with the initial learning rate adjusted to 4.5e-6 and the batch size set to 16.

[0083] (3) Use the data D1 constructed in step (1) and the two reconstructed variational autoencoders obtained in step (2) to train the conditional diffusion model to obtain the adaptive posture generation model M1.

[0084] (3.1) The adaptive posture generation model M1 is based on the U-Net network, including the clothing VAE reconstruction variational autoencoder R1 and the posture VAE reconstruction variational autoencoder R2.

[0085] (3.2) The model uses {text description, target clothing image, posture image} as training data, and the data batch size is B. The target clothing image c and posture image p are respectively encoded by encoder E in R1 and R2. R1 and E R2 Coded as z c =E R1 (c),z p =E R2 (p), both After concat operation on channel dimension, we get The result of size is used as the input of the U-Net network.

[0086] (3.3) In the forward process of the conditional diffusion model, noise ∈ is gradually added to the latent representation z. The time step t is usually t∈[0,T], T=1000. The latent variable after adding noise in step t is zt , such as the process of adding noise from step t-1 to step t is as follows:

[0087]

[0088] The Gaussian noise ∈ t are independent of each other, so from z0 to z t The noise addition can be completed in one step, as shown in the following formula:

[0089]

[0090] Among them, the above α t are all predefined hyperparameters, and α t =1-β t , The commonly used parameter setting is β t Gradually increase from 0.0001 to 0.002.

[0091] In the denoising process, a denoising network ∈ θ , predict z t The noise added in step t∈ θ (z t ,t), z t Subtract the predicted noise to get the predicted z t-1 , iterate this process, and finally gradually restore the potential representation z t For the predicted initial potential representation z0, the denoising process from step t to step t-1 is as follows:

[0092]

[0093] Among them, in order to improve the diversity of image generation in the conditional diffusion model, ∈ is the additional Gaussian noise, σ t is a hyperparameter that adjusts the noise amplitude.

[0094] For the adaptive posture generation model M1, after obtaining the denoised z0, it is decoded by R2 R2 The final generated posture p can be obtained c =D R2 (z0).

[0095] The loss function of the training process of the denoising network prediction corresponding to the time step noise in the adaptive posture generation model M1 is as follows:

[0096]

[0097] Among them, z t represents the pose encoding after adding noise in step t, z crepresents clothing code, ∈ represents noise, ∈ θ (z t ,t,z c ) is the noise prediction network, used to predict z0 to z t noise.

[0098] During the sampling process, the input is Gaussian noise, and the model needs to gradually denoise the Gaussian noise. At time step t, the model predicts the corresponding noise ∈ θ (z t ,t), z t Subtract the predicted noise to get the predicted z t-1 , iterate this process, and finally gradually restore the Gaussian noise to the predicted initial potential representation z0. After decoding z0, the posture image adapted to the clothing can be obtained. The result example is as follows Figure 3 shown.

[0099] (3.4) For the dataset used, it is divided into training data and test data with a ratio of 9:1, and the data is scaled to a size of (512, 384). The training is optimized using the AdamW optimizer, with the initial learning rate adjusted to 2.0e-06 and the batchsize set to 16.

[0100] (4) According to step (1), the area other than clothing in the model clothing display image is divided into the area to be generated, and the corresponding data D2 of {text description, clothing image, mask to be generated, image to be generated, posture image, model clothing display image} is constructed.

[0101] (4.1) The clothing segmentation mask obtained in step (1.2) is used as the mask to be generated, the clothing part has a value of 1, and the rest is the part to be generated with a value of 0. The target clothing image obtained in step (1.2) is used as the image to be generated.

[0102] (4.2) The mask to be generated and the image to be generated obtained in step (4.1) are constructed with other data to obtain corresponding data D2 of {text description, clothing image, mask to be generated, image to be generated, posture image, model clothing display image}.

[0103] (5) Using the data D2 constructed in step (4), the clothing image is spliced ​​with the image to be generated, the posture image, and the model clothing display image in the spatial dimension, and the mask to be generated is spliced ​​with a mask of the same size initialized to 0, to obtain the corresponding data D3 of {text description, combined mask to be generated, combined image to be generated, combined posture image, combined model clothing display image}.

[0104] (5.1) Using the data D2 obtained in step (4.2), the clothing image is spliced ​​with the image to be generated, the posture image, and the model clothing display image in the spatial dimension to obtain a combined image to be generated, a combined posture image, and a combined model clothing display image whose width is twice that of the original image.

[0105] (5.2) Using the mask to be generated obtained in step (4.1), initialize a mask of the same size as the mask to be generated and with values ​​of all 0, concatenate the two in the spatial dimension to obtain a combined mask to be generated with a width twice that of the original mask.

[0106] (5.3) The data obtained in step (5.1) and step (5.2) are constructed to obtain corresponding data D3 of {text description, combined mask to be generated, combined image to be generated, combined posture image, combined model clothing display image}.

[0107] (6) Using the data D3 in step (5), the conditional diffusion model is trained based on the Stable Diffusion model to obtain the clothing enhanced display image generation model M2. The model performs clothing enhancement from the two parts of feature fusion and guided generation to better retain clothing information. The result example is as follows: Figure 4 shown.

[0108] (6.1) The clothing enhanced display image generation model is based on U-Net, including VAE encoder, U-Net network, VAE decoder, and CILP text encoder. The VAE model and CLIP text encoder used here are the same as the basic model Stable Diffusion, while U-Net is different in input, intermediate results, and output.

[0109] (6.2) The model uses the data D3 in step (4.3) as training data, and uses feature fusion and guided generation to enhance clothing details; the feature fusion method used in step (6) includes: using feature space connection, concatenating the data D2 in step (4.2) with the corresponding clothing image or mask in the spatial dimension to obtain the data D3 obtained in step (5.3), and then sending it to the encoder to obtain each potential representation, and concatenating all potential representations as the input of the U-Net network; using the attention mechanism, the text features encoded by the large-scale text image pre-training model CLIP are injected into the model by cross-attention to control the global features of the generated image, and at the same time, the clothing is used as an additional condition, combined with other conditions in the spatial dimension, and the self-attention mechanism in the denoising U-Net is used. The self-attention operation can be expressed by the following formula:

[0110]

[0111] in, It is a stacked vector of feature maps from the same latent space, p represents the number of pixels in the feature map, and d represents the vector dimension. Self-attention takes into account the correlation between each pixel in the feature map. Through this operation, the texture transfer from the clothing area in the combined image to the task area can be naturally achieved. Moreover, the self-attention mechanism can establish a direct connection between any two positions in the image, capture long-distance dependencies, optimize the intersection area of ​​the target clothing and the human body, and further integrate the extracted clothing texture details with the person image.

[0112] The guided generation method used in step (6) includes: using an additional classifier-free guidance (CFG) technology to generate conditional guidance for the image. CFG calculates the residual of the prediction results of conditional generation and unconditional generation, and uses the conditional guidance coefficient to amplify the influence of the condition in the generation process, so as to achieve the effect of conditional control enhancement. The original Stable Diffusion performs CFG on the text condition through the following formula:

[0113] ∈ θ (z t ,c t )=∈ θ (z t )+S t *(∈ θ (z t ,c t )-∈ θ (z t ))

[0114] Among them, ∈ θ (z t ,c t ) represents the text condition c t The result of guiding prediction, ∈ θ (z t ) represents the result of the prediction without text condition, S t The text condition guidance coefficient is set manually;

[0115] Step (6) uses the improved CFG of the combined text condition and clothing condition to improve the generation effect of the final image. The specific formula is as follows:

[0116] ∈ θ (z t ,c t ,c c )=∈ θ (z t )+S c *(∈θ (z t ,c c )-∈ θ (z t ))+S t *(∈ θ (z,c t ,c c )

[0117] -∈ θ (z t ,c c ))

[0118] ∈ θ (x t ,c t ,c c ) represents the text condition c t With clothing conditions c c The prediction results of joint guidance, S c is the clothing condition guidance coefficient.

[0119] During the training process, 5% of the training samples are randomly set as 5% of the training samples are 5% of the training samples have

[0120] During sampling, for time step t, the model predicts the noise ∈ θ (z t ), with text condition c t Guided noise ∈ θ (z t ,c t ) and text condition c t With clothing conditions c c The jointly guided noise ∈ θ (z t ,c t ,c c ), through the pre-set text condition guidance coefficient S t and clothing condition guidance coefficient S c Calculation is performed to obtain the final predicted noise at time step t, and the predicted z0 is obtained after gradual denoising. After decoding, a preliminary virtual model clothing display image is obtained.

[0121] (6.3) In the noise adding process of the conditional diffusion model, noise ∈ is added to the latent representation z step by step. The time step t is usually t∈[0,T], T=1000. The latent variable after t-step noise addition is z t , in the denoising process, train a denoising network ∈ θ , predict z tThe noise added in step t∈ θ (z t ,t), z t Subtract the predicted noise to get the predicted z t-1 , iterate this process, and finally gradually restore the potential representation z t For the predicted initial potential representation z0, the loss function of the entire prediction training process is as follows:

[0122]

[0123] Among them, z t represents the clothing display image encoding after adding noise in step t, z c represents clothing conditional encoding, m represents the downsampled mask to be generated, x represents the input model clothing display image, ⊙ represents element-by-element multiplication, z p represents the pose conditional encoding, e represents the text feature of CLIP encoding, z m⊙x represents the conditional coding of the image to be generated, ∈ represents noise, ∈ θ (z t ,z m⊙x ,z c ,t,m,z p ,e) is the noise prediction network, used to predict z0 to z t Noise;

[0124] (6.4) For the dataset used, it is divided into training data and test data with a ratio of 9:1, and the data is scaled to a size of (512, 384). The training is optimized using the AdamW optimizer, with the initial learning rate adjusted to 1.0e-05 and the batchsize set to 20.

[0125] (7) For an image of a mannequin or any person wearing target clothing, use a pre-trained clothing segmentation model to perform segmentation to obtain the input target clothing;

[0126] (8) Input the target clothing into the adaptive posture model M1 to obtain the generated posture that adapts to the clothing.

[0127] (9) The target clothing and generated posture obtained in step (7) and step (8) are input into the clothing enhanced display image generation model M2 to obtain a preliminary virtual model clothing display image.

[0128] (10) The preliminary virtual model clothing display image obtained in step (9) is input into the pre-trained face restoration model CodeFormer to obtain the optimized final virtual model clothing display image.

[0129] (10.1) Perform face detection on the preliminary virtual model clothing display image obtained in step (9) using a pre-trained model, crop the detected face in the image, and perform fixed-size scaling.

[0130] (10.2) For the face obtained in step (10.1), the high-quality face image blocks in the pre-trained codebook are used in combination with Transformer to capture global face information, and blind face repair of low-quality or damaged facial images is achieved. The resulting multi-pose virtual model clothing display image result example is shown in the figure below. Figure 5 shown.

[0131] Corresponding to the aforementioned embodiment of a method for intelligently generating a virtual model clothing display image based on a diffusion model, the present invention also provides an embodiment of a device for intelligently generating a virtual model clothing display image based on a diffusion model.

[0132] See also Figure 6 An embodiment of the present invention provides a device for intelligently generating virtual model clothing display images based on a diffusion model, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a method for intelligently generating virtual model clothing display images based on a diffusion model in the above embodiment.

[0133] An embodiment of a device for intelligently generating virtual model clothing display images based on a diffusion model provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 6 As shown in FIG. 1 , a hardware structure diagram of a device having data processing capability in which a virtual model clothing display image intelligent generation device based on a diffusion model provided by the present invention is located, except that Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0134] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0135] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0136] An embodiment of the present invention further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a method for intelligently generating a virtual model clothing display image based on a diffusion model in the above embodiment is implemented.

[0137] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0138] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the method for intelligently generating a virtual model clothing display image based on a diffusion model is implemented.

[0139] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solution formed by equivalent replacement or equivalent transformation falls within the protection scope required by the present invention.

Claims

1. A method for intelligently generating virtual model clothing display images based on a diffusion model, characterized in that: The method comprises the following steps: (1) Obtain a paired data set of {clothing image, model clothing display image} and an unpaired data set containing only model clothing display images, and segment the target clothing image of the model clothing display image; perform pose estimation and text description extraction at the same time to obtain a data pair of {text description, target clothing image, pose image}; (2) Based on the data in step (1), the conditional diffusion model is trained to obtain an adaptive posture generation model M1; (3) Dividing the area except the clothing in the model clothing display image into the to-be-generated area, obtaining the to-be-generated image and the to-be-generated mask, splicing the clothing image with the to-be-generated image, the posture image and the model clothing display image in the spatial dimension, and splicing the to-be-generated mask with a mask of the same size initialized to 0; (4) Based on the spliced ​​image and mask training conditional diffusion model, the clothing enhanced display image generation model M2 is obtained, which is used to enhance clothing details from the two parts of feature fusion and guided generation; The feature fusion method includes: using feature space connection, concatenating data D2 with the corresponding clothing image or mask in the spatial dimension to obtain data D3, sending it to the encoder to obtain various potential representations, and concatenating all the potential representations as the input of the U-Net network; using the attention mechanism, injecting the text features encoded by the large-scale text image pre-training model CLIP into the model through cross-attention to control the global features of the generated image, and taking clothing as an additional condition, combining it with other conditions in the spatial dimension, and establishing a direct connection between any two positions in the image through the self-attention mechanism in the denoising U-Net, capturing long-distance dependencies, optimizing the intersection area of ​​the target clothing and the human body part, and further merging the extracted clothing texture details with the character image; The guided generation method includes: using an additional classifier-free guided CFG technology to guide the condition of the generated image. CFG calculates the residual of the prediction results of conditional generation and unconditional generation, and uses the conditional guidance coefficient to amplify the influence of the condition in the generation process, so as to achieve the effect of conditional control enhancement. The CFG of the combined text condition and clothing condition is used to improve the generation effect of the final image. The specific formula is as follows: ∈ θ (z t ,c t ,c c )=∈ θ (z t )+S c *(∈ θ (z t ,c c )-∈ θ (z t ))+S t *(∈ θ (z,c t ,c c ) -∈ θ (z t ,c c )) Among them, ∈ θ (z t ,c t ) represents the text condition c t The result of guiding prediction, ∈ θ (z t ) represents the result of the prediction without text condition, S t is the text conditional guidance coefficient, ∈ θ (z t ,c t ,c c ) represents the text condition c t With clothing conditions c c The prediction results of joint guidance, S c is the clothing condition guidance coefficient; (5) obtaining an image for virtual model clothing display, segmenting the target clothing image; inputting it into model M1 to obtain a generated posture image adapted to the clothing, inputting the target clothing image and the generated posture image into model M2 together to obtain a preliminary virtual model clothing display image, and obtaining the optimized final virtual model clothing display image based on the face repair model.

2. The method for intelligently generating virtual model clothing display images based on a diffusion model according to claim 1, characterized in that: The step (1) comprises the following steps: (1.1) Obtain a paired data set containing {clothing image, model clothing display image} data pairs and an unpaired data set containing only clothing display images for data processing, scale the data set to a specified size, convert the data type, and normalize the data; (1.2) Using the pre-trained clothing segmentation model on the model clothing display image, obtain the clothing segmentation mask of the original image. The size of the clothing segmentation mask is equal to the original image, and the value of the clothing part is 1, and the value of the rest is 0. The target clothing image is obtained according to the clothing segmentation mask; (1.3) Perform posture estimation and text description extraction on the dataset to obtain the posture image corresponding to the model clothing display image and construct the target clothing image obtained through step (1.2) into data D1 of {text description, target clothing image, posture image}.

3. The method for intelligently generating virtual model clothing display images based on a diffusion model according to claim 2, characterized in that: The step (2) comprises the following steps: (2.1) The VAE reconstruction variational autoencoder is an encoder-decoder structure, which includes an encoder with 22 convolution layers, 10 residual blocks and a symmetrical decoder; (2.2) VAE reconstruction variational autoencoder uses clothing images and posture images as training data, and uses them as input after fixed scaling, type conversion, and normalization; (2.3)VAE reconstruction variational autoencoder loss consists of reconstruction loss and perceptual loss; (2.4) Use the Adam optimizer for optimization and adjust the learning rate and batch size hyperparameters. After the parameters are set, start training. After the training is completed, the clothing image reconstruction variational autoencoder R1 and the posture reconstruction variational autoencoder R2 are obtained; (2.5) The adaptive posture generation model M1 is based on U-Net as the basic network, including VAE encoder, U-Net network and VAE decoder; (2.6) The adaptive posture generation model M1 uses the data D1 in step (1.3) as training data, encodes the target clothing image and posture image into the latent space, and uses them as the input of the U-Net network after the concat operation of the channel dimension; (2.7) In the noise adding process of the conditional diffusion model, noise ∈ is added to the latent representation z step by step. After time step t, the latent variable after noise addition is z t , in the denoising process, train a denoising network ∈ θ , predict z t The noise added in step t∈ θ (z t ,t), z t Subtract the predicted noise to get the predicted z t-1 , iterate this process, and finally gradually restore the potential representation z t For the predicted initial potential representation z0, the loss function of the entire training process is as follows: in, represents the mean square error loss, z t represents the pose encoding after adding noise in the tth step, z c represents clothing code, ∈ represents noise, ∈ θ (z t ,t,z c ) is the noise prediction network, used to predict z0 to z t Noise; represents the square of the L2 norm, that is, the sum of the squares of each element; (2.8) Use AdamW optimizer for optimization and adjust the learning rate and batch size hyperparameters. After the parameters are set, start training. After the training is completed, the adaptive posture generation model M1 is obtained.

4. The method for intelligently generating virtual model clothing display images based on a diffusion model according to claim 2, characterized in that: The step (3) comprises the following steps: (3.1) The clothing segmentation mask obtained in step (1.2) is used as the mask to be generated, the clothing part has a value of 1, and the rest is the part to be generated with a value of 0, and the target clothing image obtained in step (1.2) is used as the image to be generated; (3.2) The mask to be generated obtained in step (3.1) and the image to be generated are used to construct corresponding data D2 of {text description, clothing image, mask to be generated, image to be generated, posture image, model clothing display image}; (3.3) Using the data D2 obtained in step (3.2), the clothing image is spliced ​​with the image to be generated, the posture image, and the model clothing display image in the spatial dimension to obtain a combined image to be generated, a combined posture image, and a combined model clothing display image with a width twice that of the original image; (3.4) Using the mask to be generated obtained in step (3.1), initialize a mask of the same size as the mask to be generated and whose value is all 0, and concatenate the two in the spatial dimension to obtain a combined mask to be generated whose width is twice that of the original mask; (3.5) The data obtained in step (3.3) and step (3.4) are constructed to obtain corresponding data D3 of {text, combined mask to be generated, combined image to be generated, combined posture image, combined model clothing display image}.

5. The method for intelligently generating virtual model clothing display images based on a diffusion model according to claim 4, characterized in that: The step (4) comprises the following steps: (4.1) The clothing enhanced display image generation model M2 is based on U-Net as the basic network, including VAE encoder, U-Net network, VAE decoder, and CILP text encoder; (4.2) The model uses the data D3 in step (3.5) as training data, and uses feature fusion and guided generation to enhance clothing details; (4.3) In the noise adding process of the conditional diffusion model, noise ∈ is added to the latent representation z step by step. After time step t, the latent variable after noise addition is z t , in the denoising process, train a denoising network ∈ θ , predict z t The noise added in step t∈ θ (z t ,t), z t Subtract the predicted noise to get the predicted z t-1 , iterate this process, and finally gradually restore the potential representation z t For the predicted initial potential representation z0, the loss function of the entire training process is as follows: in, represents the mean square error loss, z t represents the clothing display image encoding after adding noise in step t, z c represents clothing conditional encoding, m represents the downsampled mask to be generated, x represents the input model clothing display image, ⊙ represents element-by-element multiplication, z p represents the pose conditional encoding, e represents the text feature of CLIP encoding, z m⊙x represents the conditional coding of the image to be generated, ∈ represents noise, ∈ θ (z t ,z m⊙x ,z c ,t,m,z p ,e) is the noise prediction network, used to predict z0 to z t Noise; represents the square of the L2 norm, that is, the sum of the squares of each element; (4.4) Use AdamW optimizer for optimization and adjust the learning rate and batch size hyperparameters. After the parameters are set, start training. After the training is completed, the clothing enhanced display image generation model M2 is obtained.

6. The method for intelligently generating virtual model clothing display images based on a diffusion model according to claim 1, characterized in that: The step (5) comprises the following steps: (5.1) For an image of a mannequin or any person wearing target clothing, use the pre-trained clothing segmentation model to segment the image and obtain the input target clothing; (5.2) Input the target clothing into the adaptive posture model M1 to obtain the generated posture of the adaptive clothing; (5.3) Inputting the target clothing and generated posture into the clothing enhanced display image generation model M2 to obtain a preliminary virtual model clothing display image; (5.4) Using the pre-trained model to perform face detection on the obtained preliminary virtual model clothing display image, cropping the detected face in the image, and scaling it to a fixed size; (5.5) For the obtained face, the high-quality face image blocks in the pre-trained codebook are used in combination with Transformer to capture global face information, realize blind face repair of low-quality or damaged facial images, and obtain the optimized final virtual model clothing display image.

7. A device for intelligently generating virtual model clothing display images based on a diffusion model, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a method for intelligently generating a virtual model clothing display image based on a diffusion model as described in any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a method for intelligently generating a virtual model clothing display image based on a diffusion model as described in any one of claims 1 to 6 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for intelligently generating virtual model clothing display images based on a diffusion model as described in any one of claims 1 to 6 is implemented.

Images