Virtual model clothing display image intelligent generation method and device based on diffusion model

Abstract

The invention discloses a method and a device for intelligently generating a virtual model clothing display image based on a diffusion model. Processing by using original data in the data set and the clothing segmentation model to obtain a target clothing image and a posture image, and training to obtain a reconstruction variational auto-encoder of the clothing and the posture and an adaptive posture generation model M1; performing spatial splicing construction on the original data in the data set to obtain combined data; performing fine tuning training of a conditional diffusion model by using the combined data, and performing garment enhancement from feature fusion and guide generation to obtain a garment enhanced display image generation model M2; and inputting an image of target clothes worn by a primitive mannequin model or any person, obtaining a preliminary clothes display image through M1 and M2, and obtaining a final clothes display image through face restoration model optimization. According to the invention, innovation is realized from the aspects of a training mode, clothing feature retention and a generation result, and the virtual model clothing display image which is convenient to operate and high in quality can be generated.

Description

Technical Field

[0001] The present invention relates to the field of image generation, and in particular to a method and a device for intelligently generating virtual model clothing display images based on a diffusion model. Background Art

[0002] With the development of artificial intelligence (AI), generative artificial intelligence (AIGC) technology has been able to automatically generate high-quality digital content such as images and videos. Therefore, AIGC technology has also been widely used in clothing display tasks to generate specific clothing display images. However, the current clothing display image generation technology has problems such as relying on paired data training, failing to maintain information such as clothing pattern and texture, and insufficient generation diversity. It cannot bring real wearing effects to consumers and is difficult to fully meet the needs of different merchants and consumers, limiting the wide application and further promotion of AIGC in clothing display tasks.

[0003] Most of the current clothing display image generation is based on real model clothing image generation. According to the technical context of the generative model, it can be divided into methods based on CNN or GAN. These methods approximate the deformation of clothing through geometric matching of two-dimensional images, but this kind of deformation is only obtained by mapping and twisting the image, and cannot simulate the subtle changes of clothing such as wrinkles and flutter in the real world. With the development of the Diffusion model, a method for generating real model clothing images based on the Diffusion model has gradually emerged. Although the Diffusion model has demonstrated its potential in generating high-quality images in clothing displays, when the target is complex clothing, there are still problems such as insufficient detail retention and low control accuracy. In terms of data, the diversity and detailed features of complex clothing also make it difficult to build a sufficiently comprehensive and high-quality dataset, which may lead to poor performance of the model when dealing with complex clothing that has not been seen before.

[0004] In summary, the current clothing display image generation technology is developing rapidly, and the quality of the generated results is getting higher and higher. However, most of the current clothing display methods are based on real model clothing display methods generated by existing real models. Such methods usually require paired data sets corresponding to clothing and models for training, and this method also has problems such as difficulty in retaining clothing texture details and single model features in the generated images. At the same time, because the clothing display images generated by such methods retain the existing models, merchants do not have copyrights and cannot be directly displayed commercially.

[0005] In view of the shortcomings of previous clothing display image generation methods, the present invention proposes an intelligent generation method for virtual model clothing display images based on a diffusion model. This method breaks through the limitations of paired data on model training, and while more effectively retaining clothing texture details, generates multi-pose virtual model clothing display images of matching clothing through prior target clothing. This method has more diverse generation results and allows merchants to directly conduct commercial clothing displays. Summary of the invention

[0006] In view of the above shortcomings, the present invention provides a method and device for intelligently generating virtual model clothing display images based on a diffusion model. The method only requires an image of a plain mannequin or any person wearing target clothing to generate virtual model clothing display images in various postures. The method innovatively proposes a new mode of clothing expansion. At the same time, it integrates clothing adaptive posture generation and classifier-free guidance modules to better retain clothing details, improve generation quality and diversity, and achieve higher quality virtual model clothing display image generation.

[0007] In order to achieve the above-mentioned object, the technical solution of the present invention is as follows: In a first aspect, the present invention provides a method for intelligently generating a virtual model clothing display image based on a diffusion model, the method comprising the following steps: (1) Obtain a paired dataset of {clothing image, model clothing display image} and an unpaired dataset containing only model clothing display images, and segment the target clothing image from the model clothing display image; perform pose estimation and text description extraction at the same time to obtain a data pair of {text description, target clothing image, pose image}; (2) Based on the data in step (1), the conditional diffusion model is trained to obtain the adaptive posture generation model M1; (3) Divide the area except clothing in the model clothing display image into the to-be-generated area, obtain the to-be-generated image and the to-be-generated mask, splice the clothing image with the clothing image, the to-be-generated image, the posture image and the model clothing display image in the spatial dimension, and splice the to-be-generated mask with the mask of the same size initialized to 0; (4) Based on the spliced ​​image and mask, the conditional diffusion model is trained to obtain the clothing enhanced display image generation model M2, which is used to enhance clothing details from the two parts of feature fusion and guided generation; (5) Obtain an image that needs to be displayed on a virtual model, segment the target clothing image, input it into model M1, obtain a generated posture image adapted to the clothing, input the target clothing image and the generated posture image into model M2 together, obtain a preliminary virtual model clothing display image, and obtain the optimized final virtual model clothing display image based on the face repair model.

[0008] Furthermore, the step (1) comprises the following steps: (1.1) Obtain a paired data set containing {clothing images, model clothing display images} data pairs and an unpaired data set containing only clothing display images for data processing, scale the data set to a specified size, convert the data type, and normalize the data; (1.2) Use the pre-trained clothing segmentation model on the model clothing display image to obtain the clothing segmentation mask of the original image. The size of the clothing segmentation mask is equal to the original image, and the value of the clothing part is 1, and the value of the rest is 0. The target clothing image is obtained based on the clothing segmentation mask; (1.3) Perform pose estimation and text description extraction on the dataset to obtain the pose image corresponding to the model clothing display image and construct the target clothing image obtained through step (1.2) into data D1 of {text description, target clothing image, pose image}.

[0009] Furthermore, the step (2) comprises the following steps: (2.1) The VAE reconstruction variational autoencoder is an encoder-decoder structure, which includes an encoder with 22 convolution layers, 10 residual blocks and a symmetrical decoder; (2.2) VAE reconstruction variational autoencoder uses clothing images and posture images as training data, and then performs fixed scaling, type conversion, and normalization as input; (2.3) VAE reconstruction variational autoencoder loss consists of reconstruction loss and perceptual loss; (2.4) Use the Adam optimizer for optimization and adjust the learning rate and batch size hyperparameters. After the parameters are set, start training. After the training is completed, obtain the clothing image reconstruction variational autoencoder R1 and the posture reconstruction variational autoencoder R2; (2.5) The adaptive posture generation model M1 is based on U-Net, including VAE encoder, U-Net network and VAE decoder; (2.6) The adaptive posture generation model M1 uses the data D1 in step (1.3) as training data, encodes the target clothing image and posture image into the latent space, and uses them as the input of the U-Net network after the concat operation of the channel dimension; (2.7) In the noise addition process of the conditional diffusion model, the latent representation Add noise gradually , after a time step The latent variable after adding noise is , in the denoising process, train a denoising network , predict exist Step-added noise ,Will Subtract the predicted noise to get the predicted , iterate this process and gradually restore the potential representation is the initial latent representation for prediction , the loss function of the entire training process is as follows:

[0010] in, represents the mean square error loss, Indicates The pose encoding after step noise addition, Indicates clothing code, represents noise, is a noise prediction network, used to predict arrive Noise; represents the square of the L2 norm, that is, the sum of the squares of each element; (2.8) Use the AdamW optimizer for optimization and adjust the learning rate and batch size hyperparameters. After the parameters are set, start training. After the training is completed, the adaptive posture generation model M1 is obtained.

[0011] Furthermore, the step (3) comprises the following steps: (3.1) The clothing segmentation mask obtained in step (1.2) is used as the mask to be generated, the clothing part has a value of 1, and the rest is the part to be generated with a value of 0. The target clothing image obtained in step (1.2) is used as the image to be generated; (3.2) The mask to be generated obtained in step (3.1) and the image to be generated are used to construct corresponding data D2 of {text description, clothing image, mask to be generated, image to be generated, posture image, model clothing display image}; (3.3) Using the data D2 obtained in step (3.2), the clothing image is spliced ​​with the clothing image, the image to be generated, the posture image, and the model clothing display image in the spatial dimension to obtain a combined clothing image, a combined image to be generated, a combined posture image, and a combined model clothing display image, each of which has a width twice that of the original image; (3.4) Using the mask to be generated obtained in step (3.1), initialize a mask of the same size as the mask to be generated with values ​​of all 0, concatenate the two in the spatial dimension, and obtain a combined mask to be generated with a width twice that of the original mask; (3.5) The data obtained in step (3.3) and step (3.4) are constructed to obtain corresponding data D3 of {text, combined clothing image, combined mask to be generated, combined image to be generated, combined posture image, combined model clothing display image}.

[0012] Furthermore, the step (4) comprises the following steps: (4.1) The clothing enhanced display image generation model M2 is based on the U-Net network, including VAE encoder, U-Net network, VAE decoder, and CILP text encoder; (4.2) The model uses the data D3 in step (3.5) as training data, and uses feature fusion and guided generation to enhance clothing details; (4.3) In the noise addition process of the conditional diffusion model, the latent representation Add noise gradually , after a time step The latent variable after adding noise is , in the denoising process, train a denoising network , predict exist Step-added noise ,Will Subtract the predicted noise to get the predicted , iterate this process and gradually restore the potential representation is the initial latent representation for prediction , the loss function of the entire training process is as follows:

[0013] in, represents the mean square error loss, Indicates The clothing display image encoding after step noise addition, Indicates clothing condition code, represents the mask to be generated for downsampling, Indicates the input model clothing display image, represents element-wise multiplication, represents the posture conditional encoding, represents the text features encoded by CLIP, Represents the conditional encoding of the image to be generated, represents noise, is a noise prediction network, used to predict arrive Noise; represents the square of the L2 norm, that is, the sum of the squares of each element; (4.4) Use the AdamW optimizer for optimization and adjust the learning rate and batch size hyperparameters. After the parameters are set, start training. After the training is completed, the clothing enhanced display image generation model M2 is obtained.

[0014] Furthermore, the feature fusion method used in step (4.2) includes: using feature space connection, concatenating data D2 with the corresponding clothing image or mask in the spatial dimension to obtain data D3, and then sending it to the encoder to obtain each potential representation, and concatenating all the potential representations as the input of the U-Net network; using the attention mechanism, injecting the text features encoded by the large-scale text image pre-training model CLIP into the model through cross-attention to control the global features of the generated image, and taking clothing as an additional condition, combining it with other conditions in the spatial dimension, and establishing a direct connection between any two positions of the image through the self-attention mechanism in the denoising U-Net, capturing long-distance dependencies, optimizing the intersection area of ​​the target clothing and the human body part, and further merging the extracted clothing texture details with the character image; The guided generation method used in step (4.2) includes: using an additional classifier-free guided CFG technology to generate conditional guidance of the image. CFG calculates the residual of the conditional generation and unconditional generation prediction results, and uses the conditional guidance coefficient to amplify the influence of the condition in the generation process, so as to achieve the effect of conditional control enhancement. Step (4.2) uses an improved CFG that uses joint text conditions and clothing conditions to improve the generation effect of the final image. The specific formula is as follows:

[0015] in, Indicates text conditions The results of the guided prediction, represents the result of the no-text conditional guidance prediction, is the text conditional guidance coefficient, Indicates text conditions Clothing conditions The prediction results of joint guidance, is the clothing condition guidance coefficient.

[0016] Furthermore, the step (5) comprises the following steps: (5.1) For an image of a mannequin or any person wearing target clothing, use the pre-trained clothing segmentation model to segment the image and obtain the input target clothing; (5.2) Input the target clothing into the adaptive posture model M1 to obtain the generated posture adapted to the clothing; (5.3) Input the target clothing and generated posture into the clothing enhanced display image generation model M2 to obtain a preliminary virtual model clothing display image; (5.4) Using the pre-trained model to perform face detection on the obtained preliminary virtual model clothing display image, cropping the detected face in the image, and scaling it to a fixed size; (5.5) For the obtained face, use the high-quality face image blocks in the pre-trained codebook and combine them with Transformer to capture global face information, realize blind face repair of low-quality or damaged facial images, and obtain the optimized final virtual model clothing display image.

[0017] In a second aspect, the present invention also provides a device for intelligently generating virtual model clothing display images based on a diffusion model, comprising a memory and one or more processors, wherein the memory stores executable code, and wherein when the processor executes the executable code, it implements the method for intelligently generating virtual model clothing display images based on a diffusion model.

[0018] In a third aspect, the present invention further provides a computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the method for intelligently generating virtual model clothing display images based on a diffusion model is implemented.

[0019] In a fourth aspect, the present invention further provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the method for intelligently generating virtual model clothing display images based on a diffusion model.

[0020] The beneficial effects of the present invention are as follows: only one image of a plain mannequin or any person wearing target clothing is needed to generate virtual model clothing display images with various postures; the present method innovatively proposes a new clothing expansion generation mode, and the model can construct a (clothing, model image) data pair by itself through a clothing segmentation model according to a single clothing display image. This method greatly increases the available data, so that the model does not need to rely on paired data for training and is not constrained by data scarcity; the present method innovatively combines feature fusion and guided enhancement to enhance clothing details, and maximizes the retention of clothing pattern, texture details and other features through feature space connection, attention mechanism and conditional guidance of joint text conditions and clothing conditions; the present method innovatively combines an adaptive posture generation module, and through additional posture information, the model generation result has a better human body structure, and the final virtual model has a variety of postures, and the virtual model clothing display image generation method of the present invention can generate virtual models that do not exist in the world, avoid copyright disputes, and e-commerce merchants can directly use the generated images for online clothing display. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The present invention provides a flow chart of a method for intelligently generating virtual model clothing display images based on a diffusion model.

[0022] Figure 2 This is an example of the result of clothing segmentation.

[0023] Figure 3 This is an example of the results of the adaptive posture model.

[0024] Figure 4 This is an example of the results of the clothing enhanced display image generation model.

[0025] Figure 5 This is an example of the generated multi-pose virtual model clothing display image results.

[0026] Figure 6 It is a structural diagram of a device for intelligently generating virtual model clothing display images based on a diffusion model provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, the present invention is further described below with reference to the accompanying drawings.

[0028] Aiming at the task of generating clothing display images for practical applications, the present invention proposes an intelligent generation method for virtual model clothing display images based on a diffusion model. The method can generate virtual model clothing display images with multiple postures through an image of a human body model or any person wearing target clothing. After clothing segmentation of a single input clothing display image, the target clothing is obtained, which can form a {clothing image, model clothing display image} data pair with the input clothing display image, which makes the method break through the shackles of traditional real model clothing display images that can only be trained using paired data. Considering that the human body structure generation effect is not good when only using the target clothing as input to directly generate clothing display images, an adaptive posture generation module is added. By taking the target clothing as input, multiple different postures adapted to clothing are generated, and the posture is used as additional information to guide the generation of the human body structure of the clothing display image, and the diversity of the clothing display image generation results is also improved. In order to further strengthen the retention of clothing pattern, texture and other detail information, the method also performs clothing enhancement from two parts: feature fusion and guided generation. Finally, a pre-trained face repair model is used to improve the quality of human body details in the final virtual model clothing display image.

[0029] Example 1 like Figure 1 As shown, an embodiment of the present invention provides a method for intelligently generating a virtual model clothing display image based on a diffusion model, and the specific steps are as follows: (1) The paired dataset Dresscode containing data pairs of {clothing image, model clothing display image} and the unpaired dataset Deepfashion containing only clothing display images are preprocessed. The target clothing images of the model clothing display images are segmented using the pre-trained clothing segmentation model, and pose estimation and text description extraction are performed to construct the corresponding data D1 of {text description, target clothing image, pose image}.

[0030] (1.1) Scale all data in the dataset to (512, 384) size, convert it to tensor format and normalize it to [-1, 1].

[0031] (1.2) Use the pre-trained clothing segmentation model huggingface-cloth-segmentation on the model clothing display image to obtain the clothing segmentation mask of the original image. The size of the clothing segmentation mask is equal to the original image, and the value of the clothing part is 1, and the value of the rest is 0. Multiply the clothing segmentation mask with the model clothing display image to obtain the target clothing image, as shown in Figure 2 shown.

[0032] (1.3) Use the pre-trained human pose estimation model and BLIP text description extraction model on the model clothing display images in the dataset to obtain the pose image and text description of the original image.

[0033] (1.4) The posture image, text description and target clothing image obtained by step (1.2) corresponding to the model clothing display image in the data set are constructed into {text description, target clothing image, posture image} data D1, where each pair of target clothing image and posture image corresponds to the same model clothing display image.

[0034] (2) Use the data D1 constructed in step (1) to train the VAE reconstruction variational autoencoder of the target clothing image and posture image, and obtain the clothing image reconstruction variational autoencoder R1 and the posture reconstruction variational autoencoder R2.

[0035] (2.1) The model structure of the VAE reconstruction variational autoencoder is an encoder-decoder structure, which includes an encoder with 22 convolution layers, 10 residual blocks and a symmetrical decoder.

[0036] (2.2) The model uses {target clothing image, pose image} as training data, which is scaled to (512, 384) size, transformed into a tensor, and normalized to [-1, 1].

[0037] (2.3) VAE reconstruction variational autoencoder loss consists of reconstruction loss and perceptual loss. The reconstruction loss is L1loss. Specifically, for the number of pixels The input image The reconstruction result is , then the L1 loss here can be expressed as:

[0038] in represents the absolute value, and Represent the i-th pixel of the input image and the reconstructed result image respectively.

[0039] Perceptual loss using pre-trained deep convolutional neural networks Get some feature representations of the image and use the Euclidean distance to measure the difference between these features of the original image and the reconstructed image. and the reconstruction results ,The perceptual loss here can be expressed as:

[0040] in, represents the number of layers in the network, Indicates that the image is on the network The feature dimension size of the layer, Represents the square of the L2 norm, that is, the sum of the squares of each element.

[0041] The final loss function is defined as:

[0042] (2.4) For the dataset used, it is divided into training data and test data in a ratio of 9:1, and the data is scaled to a size of (512, 384). The training is optimized using the Adam optimizer, with the initial learning rate adjusted to 4.5e-6 and the batchsize set to 16.

[0043] (3) Use the data D1 constructed in step (1) and the two reconstructed variational autoencoders obtained in step (2) to train the conditional diffusion model to obtain the adaptive posture generation model M1.

[0044] (3.1) The adaptive posture generation model M1 is based on the U-Net network, including clothing VAE reconstruction variational autoencoder R1 and posture VAE reconstruction variational autoencoder R2.

[0045] (3.2) The model uses {text description, target clothing image, pose image} as training data, and the data batch size is , the target clothing image With pose image Use the encoders in R1 and R2 respectively and Encoded as , , both sizes are , after concat operation of channel dimension, we get The result of the size is used as the input of the U-Net network.

[0046] (3.3) In the forward process of the conditional diffusion model, the latent representation Add noise gradually , time step Usually ,exist The latent variable after adding noise is , such as from Step noise to The steps are as follows:

[0047] The Gaussian noise sampled at each step are independent of each other, so arrive The noise addition can be completed in one step, as shown in the following formula:

[0048] Among them, the above are a set of predefined hyperparameters, and , , the commonly used parameter settings are Gradually increase from 0.0001 to 0.002.

[0049] In the denoising process, a denoising network is trained , predict exist Step-added noise ,Will Subtract the predicted noise to get the predicted , iterate this process and gradually restore the potential representation is the initial latent representation for prediction , as follows from Step to The denoising process of the first step:

[0050] Among them, in order to improve the diversity of image generation of the conditional diffusion model, is the additional Gaussian noise, is a hyperparameter that adjusts the noise amplitude.

[0051] For the adaptive posture generation model M1, the denoised After that, through the decoder of R2 The final generated posture can be obtained .

[0052] The loss function of the training process of the denoising network prediction corresponding to the time step noise in the adaptive posture generation model M1 is as follows:

[0053] in, Indicates The pose encoding after step noise addition, Indicates clothing code, represents noise, is a noise prediction network, used to predict arrive noise.

[0054] During the sampling process, the input is Gaussian noise, and the model needs to gradually denoise the Gaussian noise. When , the model predicts the corresponding noise ,Will Subtract the predicted noise to get the predicted , iterate this process, and finally gradually restore Gaussian noise to the initial potential representation of the prediction ,right After decoding, the posture image adapted to the clothing can be obtained. The result example is as follows Figure 3 shown.

[0055] (3.4) For the dataset used, it is divided into training data and test data in a ratio of 9:1, and the data is scaled to a size of (512, 384). The training is optimized using the AdamW optimizer, with the initial learning rate adjusted to 2.0e-06 and the batchsize set to 16.

[0056] (4) According to step (1), the area other than the clothing in the model clothing display image is divided into the area to be generated, and the corresponding data D2 of {text description, clothing image, mask to be generated, image to be generated, posture image, model clothing display image} is constructed.

[0057] (4.1) The clothing segmentation mask obtained in step (1.2) is used as the mask to be generated, the clothing part has a value of 1, and the rest is the part to be generated with a value of 0. The target clothing image obtained in step (1.2) is used as the image to be generated.

[0058] (4.2) The mask to be generated and the image to be generated obtained in step (4.1) are constructed with other data to obtain corresponding data D2 of {text description, clothing image, mask to be generated, image to be generated, posture image, model clothing display image}.

[0059] (5) Using the data D2 constructed in step (4), the clothing image is concatenated with the clothing image, the image to be generated, the posture image, and the model clothing display image in the spatial dimension, and the mask to be generated is concatenated with a mask of the same size initialized to 0, to obtain the corresponding data D3 of {text description, combined clothing image, combined mask to be generated, combined image to be generated, combined posture image, combined model clothing display image}.

[0060] (5.1) Using the data D2 obtained in step (4.2), the clothing image is spliced ​​with the clothing image, the image to be generated, the posture image, and the model clothing display image in the spatial dimension to obtain a combined clothing image with a width twice that of the original image, the image to be generated, the posture image, and the model clothing display image.

[0061] (5.2) Using the mask to be generated obtained in step (4.1), initialize a mask of the same size as the mask to be generated with values ​​of all 0, concatenate the two in the spatial dimension, and obtain a combined mask to be generated with a width twice that of the original mask.

[0062] (5.3) The data obtained in step (5.1) and step (5.2) are constructed to obtain the corresponding data D3 of {text description, combined clothing image, combined mask to be generated, combined image to be generated, combined posture image, combined model clothing display image}.

[0063] (6) Using the data D3 in step (5), the conditional diffusion model is trained based on the Stable Diffusion model to obtain the clothing enhanced display image generation model M2. This model performs clothing enhancement from two parts: feature fusion and guided generation to better retain clothing information. The results are shown in the following example. Figure 4 shown.

[0064] (6.1) The clothing enhanced display image generation model is based on U-Net, including VAE encoder, U-Net network, VAE decoder, and CILP text encoder. The VAE model and CLIP text encoder used here are the same as the basic model Stable Diffusion, while U-Net is different in input, intermediate results, and output.

[0065] (6.2) The model uses the data D3 in step (4.3) as training data, and uses feature fusion and guided generation to enhance clothing details; the feature fusion method used in step (6) includes: using feature space connection, concat the data D2 in step (4.2) with the corresponding clothing image or mask in the spatial dimension to obtain the data D3 obtained in step (5.3), and then send it to the encoder to obtain each potential representation, and concat all the potential representations as the input of the U-Net network; using the attention mechanism, the text features encoded by the large-scale text image pre-training model CLIP are injected into the model by cross-attention to control the global features of the generated image, and at the same time, the clothing is used as an additional condition, combined with other conditions in the spatial dimension, through the self-attention mechanism in the denoising U-Net, the self-attention operation can be expressed by the following formula:

[0066] in, is a stacked vector of feature maps from the same latent space, Represents the number of pixels in the feature map, Represents the vector dimension. Self-attention takes into account the correlation between each pixel in the feature map. Through this operation, the texture transfer from the clothing area in the combined image to the task area can be naturally achieved. Moreover, the self-attention mechanism can establish a direct connection between any two positions in the image, capture long-distance dependencies, optimize the intersection area of ​​the target clothing and the human body, and further integrate the extracted clothing texture details with the person image. The guided generation method used in step (6) includes: using an additional classifier-free guidance (CFG) technology to generate conditional guidance for the image. CFG calculates the residual of the prediction results of conditional generation and unconditional generation, and uses the conditional guidance coefficient to amplify the influence of the condition in the generation process, so as to achieve the effect of conditional control enhancement. The original Stable Diffusion performs CFG on the text condition through the following formula:

[0067] in, Indicates text conditions The results of the guided prediction, represents the result of the text-free conditional guidance prediction, The text condition guidance coefficient is set manually; Step (6) uses the improved CFG of the combined text condition and clothing condition to improve the generation effect of the final image. The specific formula is as follows:

[0068] Indicates text conditions Clothing conditions The prediction results of joint guidance, is the clothing condition guidance coefficient.

[0069] During the training process, 5% of the training samples are randomly set as , 5% of the training samples are , 5% of the training samples have , .

[0070] During the sampling process, for the time step , the model predicts the noise without text condition guidance , with text conditions Guided noise and text conditions Clothing conditions Jointly guided noise , through the pre-set text conditional guidance coefficient and clothing condition guidance coefficient Calculate and get the final result at time step The prediction noise is gradually removed and the predicted ,After decoding, we get the preliminary virtual model clothing display image.

[0071] (6.3) In the noise addition process of the conditional diffusion model, the latent representation Add noise gradually , time step Usually , The latent variable after adding noise is , in the denoising process, train a denoising network , predict exist Step-added noise ,Will Subtract the predicted noise to get the predicted , iterate this process and gradually restore the potential representation is the initial latent representation for prediction , the loss function of the entire prediction training process is as follows:

[0072] in, Indicates The clothing display image encoding after step noise addition, Indicates clothing condition code, represents the mask to be generated for downsampling, Indicates the input model clothing display image, represents element-wise multiplication, represents the posture conditional encoding, represents the text features encoded by CLIP, Represents the conditional encoding of the image to be generated, represents noise, is a noise prediction network, used to predict arrive Noise; (6.4) For the dataset used, it is divided into training data and test data in a ratio of 9:1, and the data is scaled to a size of (512, 384). The training is optimized using the AdamW optimizer, with the initial learning rate adjusted to 1.0e-05 and the batchsize set to 20.

[0073] (7) For an image of a mannequin or any person wearing target clothing, use the pre-trained clothing segmentation model to perform segmentation to obtain the input target clothing; (8) Input the target clothing into the adaptive posture model M1 to obtain the generated posture that adapts to the clothing.

[0074] (9) The target clothing and generated posture obtained in step (7) and step (8) are input into the clothing enhanced display image generation model M2 to obtain a preliminary virtual model clothing display image.

[0075] (10) The preliminary virtual model clothing display image obtained in step (9) is input into the pre-trained face restoration model CodeFormer to obtain the optimized final virtual model clothing display image.

[0076] (10.1) Perform face detection on the preliminary virtual model clothing display image obtained in step (9) using the pre-trained model, crop the detected face in the image, and perform fixed-size scaling.

[0077] (10.2) For the face obtained in step (10.1), the high-quality face image blocks in the pre-trained codebook are used in combination with Transformer to capture global face information, and blind face repair is achieved for low-quality or damaged facial images. The resulting multi-pose virtual model clothing display image example is shown in the figure below. Figure 5 shown.

[0078] Corresponding to the aforementioned embodiment of a method for intelligently generating a virtual model clothing display image based on a diffusion model, the present invention also provides an embodiment of a device for intelligently generating a virtual model clothing display image based on a diffusion model.

[0079] See also Figure 6 An embodiment of the present invention provides a device for intelligently generating virtual model clothing display images based on a diffusion model, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a method for intelligently generating virtual model clothing display images based on a diffusion model in the above embodiment.

[0080] An embodiment of a device for intelligently generating virtual model clothing display images based on a diffusion model provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 6 As shown in FIG. 1 , a hardware structure diagram of a device having data processing capability in which a virtual model clothing display image intelligent generation device based on a diffusion model provided by the present invention is located, except that Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0081] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0082] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0083] An embodiment of the present invention further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a method for intelligently generating a virtual model clothing display image based on a diffusion model in the above embodiment is implemented.

[0084] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0085] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the method for intelligently generating a virtual model clothing display image based on a diffusion model is implemented.

[0086] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solution formed by equivalent replacement or equivalent transformation falls within the protection scope required by the present invention.

Claims

1. A method for intelligently generating virtual model clothing display images based on a diffusion model, characterized in that: The method comprises the following steps: (1) Obtain a paired dataset of {clothing image, model clothing display image} and an unpaired dataset containing only model clothing display images, and segment the target clothing image from the model clothing display image; perform pose estimation and text description extraction at the same time to obtain a data pair of {text description, target clothing image, pose image}; (2) Based on the data in step (1), the conditional diffusion model is trained to obtain the adaptive posture generation model M1; (3) Divide the area except clothing in the model clothing display image into the to-be-generated area, obtain the to-be-generated image and the to-be-generated mask, splice the clothing image with the clothing image, the to-be-generated image, the posture image and the model clothing display image in the spatial dimension, and splice the to-be-generated mask with the mask of the same size initialized to 0; (4) Based on the spliced ​​image and mask, the conditional diffusion model is trained to obtain the clothing enhanced display image generation model M2, which is used to enhance clothing details from the two parts of feature fusion and guided generation; (5) Obtain an image that needs to be displayed on a virtual model, segment the target clothing image, input it into model M1, obtain a generated posture image adapted to the clothing, input the target clothing image and the generated posture image into model M2 together, obtain a preliminary virtual model clothing display image, and obtain the optimized final virtual model clothing display image based on the face repair model.

2. The method for intelligently generating virtual model clothing display images based on a diffusion model according to claim 1, characterized in that: The step (1) comprises the following steps: (1.1) Obtain a paired data set containing {clothing images, model clothing display images} data pairs and an unpaired data set containing only clothing display images for data processing, scale the data set to a specified size, convert the data type, and normalize the data; (1.2) Use the pre-trained clothing segmentation model on the model clothing display image to obtain the clothing segmentation mask of the original image. The size of the clothing segmentation mask is equal to the original image, and the value of the clothing part is 1, and the value of the rest is 0. The target clothing image is obtained based on the clothing segmentation mask; (1.3) Perform pose estimation and text description extraction on the dataset to obtain the pose image corresponding to the model clothing display image and construct the target clothing image obtained through step (1.2) into data D1 of {text description, target clothing image, pose image}.

3. The method for intelligently generating virtual model clothing display images based on a diffusion model according to claim 2, characterized in that: The step (2) comprises the following steps: (2.1) The VAE reconstruction variational autoencoder is an encoder-decoder structure, which includes an encoder with 22 convolution layers, 10 residual blocks and a symmetrical decoder; (2.2) VAE reconstruction variational autoencoder uses clothing images and posture images as training data, and then performs fixed scaling, type conversion, and normalization as input; (2.3) VAE reconstruction variational autoencoder loss consists of reconstruction loss and perceptual loss; (2.4) Use the Adam optimizer for optimization and adjust the learning rate and batch size hyperparameters. After the parameters are set, start training. After the training is completed, obtain the clothing image reconstruction variational autoencoder R1 and the posture reconstruction variational autoencoder R2; (2.5) The adaptive posture generation model M1 is based on U-Net, including VAE encoder, U-Net network and VAE decoder; (2.6) The adaptive posture generation model M1 uses the data D1 in step (1.3) as training data, encodes the target clothing image and posture image into the latent space, and uses them as the input of the U-Net network after the concat operation of the channel dimension; (2.7) In the noise addition process of the conditional diffusion model, the latent representation Add noise gradually , after a time step The latent variable after adding noise is , in the denoising process, train a denoising network , predict exist Step-added noise ,Will Subtract the predicted noise to get the predicted , iterate this process and gradually restore the potential representation is the initial latent representation for prediction , the loss function of the entire training process is as follows: ； in, represents the mean square error loss, Indicates The pose encoding after step noise addition, Indicates clothing code, represents noise, is a noise prediction network, used to predict arrive Noise; represents the square of the L2 norm, that is, the sum of the squares of each element; (2.8) Use the AdamW optimizer for optimization and adjust the learning rate and batch size hyperparameters. After the parameters are set, start training. After the training is completed, the adaptive posture generation model M1 is obtained.

4. The method for intelligently generating virtual model clothing display images based on a diffusion model according to claim 2, characterized in that: The step (3) comprises the following steps: (3.1) The clothing segmentation mask obtained in step (1.2) is used as the mask to be generated, the clothing part has a value of 1, and the rest is the part to be generated with a value of 0. The target clothing image obtained in step (1.2) is used as the image to be generated; (3.2) The mask to be generated obtained in step (3.1) and the image to be generated are used to construct corresponding data D2 of {text description, clothing image, mask to be generated, image to be generated, posture image, model clothing display image}; (3.3) Using the data D2 obtained in step (3.2), the clothing image is spliced ​​with the clothing image, the image to be generated, the posture image, and the model clothing display image in the spatial dimension to obtain a combined clothing image, a combined image to be generated, a combined posture image, and a combined model clothing display image, each of which has a width twice that of the original image; (3.4) Using the mask to be generated obtained in step (3.1), initialize a mask of the same size as the mask to be generated with values ​​of all 0, concatenate the two in the spatial dimension, and obtain a combined mask to be generated with a width twice that of the original mask; (3.5) The data obtained in step (3.3) and step (3.4) are constructed to obtain corresponding data D3 of {text, combined clothing image, combined mask to be generated, combined image to be generated, combined posture image, combined model clothing display image}.

5. The method for intelligently generating virtual model clothing display images based on a diffusion model according to claim 4, characterized in that: The step (4) comprises the following steps: (4.1) The clothing enhanced display image generation model M2 is based on the U-Net network, including VAE encoder, U-Net network, VAE decoder, and CILP text encoder; (4.2) The model uses the data D3 in step (3.5) as training data, and uses feature fusion and guided generation to enhance clothing details; (4.3) In the noise addition process of the conditional diffusion model, the latent representation Add noise gradually , after a time step The latent variable after adding noise is , in the denoising process, train a denoising network , predict exist Step-added noise ,Will Subtract the predicted noise to get the predicted , iterate this process and gradually restore the potential representation is the initial latent representation for prediction , the loss function of the entire training process is as follows: ； in, represents the mean square error loss, Indicates The clothing display image encoding after step noise addition, Indicates clothing condition code, represents the mask to be generated for downsampling, Indicates the input model clothing display image, represents element-wise multiplication, represents the posture conditional encoding, represents the text features encoded by CLIP, Represents the conditional encoding of the image to be generated, represents noise, is a noise prediction network, used to predict arrive Noise; represents the square of the L2 norm, that is, the sum of the squares of each element; (4.4) Use the AdamW optimizer for optimization and adjust the learning rate and batch size hyperparameters. After the parameters are set, start training. After the training is completed, the clothing enhanced display image generation model M2 is obtained.

6. The method for intelligently generating virtual model clothing display images based on a diffusion model according to claim 5, characterized in that: The feature fusion method used in step (4.2) includes: using feature space connection, concatenating data D2 with the corresponding clothing image or mask in the spatial dimension to obtain data D3, and then sending it to the encoder to obtain each potential representation, and concatenating all the potential representations as the input of the U-Net network; using the attention mechanism, injecting the text features encoded by the large-scale text image pre-training model CLIP into the model through cross-attention to control the global features of the generated image, and taking clothing as an additional condition, combining it with other conditions in the spatial dimension, and establishing a direct connection between any two positions in the image through the self-attention mechanism in the denoising U-Net, capturing long-distance dependencies, optimizing the intersection area of ​​the target clothing and the human body part, and further merging the extracted clothing texture details with the character image; The guided generation method used in step (4.2) includes: using an additional classifier-free guided CFG technology to generate conditional guidance of the image. CFG calculates the residual of the conditional generation and unconditional generation prediction results, and uses the conditional guidance coefficient to amplify the influence of the condition in the generation process, so as to achieve the effect of conditional control enhancement. Step (4.2) uses an improved CFG that uses joint text conditions and clothing conditions to improve the generation effect of the final image. The specific formula is as follows: ； in, Indicates text conditions The results of the guided prediction, represents the result of the no-text conditional guidance prediction, is the text conditional guidance coefficient, Indicates text conditions Clothing conditions The prediction results of joint guidance, is the clothing condition guidance coefficient.

7. The method for intelligently generating virtual model clothing display images based on a diffusion model according to claim 1, characterized in that: The step (5) comprises the following steps: (5.1) For an image of a mannequin or any person wearing target clothing, use the pre-trained clothing segmentation model to segment the image and obtain the input target clothing; (5.2) Input the target clothing into the adaptive posture model M1 to obtain the generated posture adapted to the clothing; (5.3) Input the target clothing and generated posture into the clothing enhanced display image generation model M2 to obtain a preliminary virtual model clothing display image; (5.4) Using the pre-trained model to perform face detection on the obtained preliminary virtual model clothing display image, cropping the detected face in the image, and scaling it to a fixed size; (5.5) For the obtained face, use the high-quality face image blocks in the pre-trained codebook and combine them with Transformer to capture global face information, realize blind face repair of low-quality or damaged facial images, and obtain the optimized final virtual model clothing display image.

8. A device for intelligently generating virtual model clothing display images based on a diffusion model, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a method for intelligently generating a virtual model clothing display image based on a diffusion model as described in any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a method for intelligently generating a virtual model clothing display image based on a diffusion model as described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for intelligently generating virtual model clothing display images based on a diffusion model as described in any one of claims 1 to 7 is implemented.

Images