Multi-modal size sensing virtual fitting method based on potential diffusion model

Abstract

The invention discloses a multi-modal size perception virtual fitting method based on a potential diffusion model. The method comprises the step of forming different-size virtual fitting models based on images through garment size perception, adaptive mask generation, garment fitting denoising, garment learning and multiple joint blocks. The garment size is adjusted through text prompt, the reality sense of the generated visual effect is enhanced, a garment size sensing module is designed, garment size information is coded and transmitted through text description guidance, a self-adaptive mask generation module is provided, and accurate garment size guidance is provided according to the size information. In order to accurately retain clothing details, a plurality of joint blocks are provided, and an image generation process is guided from a plurality of features to realize a vivid try-on effect.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual try-on, and in particular to a multi-modal size-aware virtual try-on method based on a potential diffusion model. Background Art

[0002] Image-based virtual try-on tasks aim to maintain the attributes of the target person and the texture details of the clothing, so that a given garment looks realistic on the target person. It has greatly promoted the development of e-commerce and online shopping by changing consumers' shopping habits and reducing costs, and has attracted widespread attention from academia and industry.

[0003] Traditional image-based virtual try-on methods employ generative adversarial networks (GANs) to deform clothing and align the distorted clothing with the pose of the target person. Recent studies have increasingly focused on latent diffusion models (LDMs), which significantly improve the realism of generated clothing. Compared with GAN-based virtual try-on methods, diffusion-based methods produce more realistic clothing effects and have advantages in handling complex poses and backgrounds. They introduce conditional controls to generate images at specified mask locations without changing other information. In addition, diffusion models implicitly distort clothing, thereby improving performance without the need for a separate deformation process.

[0004] Despite significant progress in diffusion-based virtual try-on methods, there is little work that takes into account clothing size information. This results in the tried-on clothing being as suitable for the target population as the original clothing, and failing to provide size reference information for users. Incorrect sizing is one of the main reasons for returns in online stores. Providing size adjustment capabilities allows users to see if different sizes of the same clothing fit them, which helps make more informed purchasing decisions. In addition, it can significantly reduce the return rate, save costs for retailers, and improve customer satisfaction. COTTON is one of the few efforts to achieve clothing size adjustment to fit the body. However, its performance is easily affected by pre-processing results.

[0005] In view of the above problems, the present invention provides a multimodal size-aware virtual try-on method based on a latent diffusion model that can generate realistic and controllable results. Summary of the invention

[0006] The present invention provides a multimodal size-aware virtual try-on method based on a latent diffusion model. The method comprises clothing size perception, adaptive mask generation, clothing try-on denoising, clothing learning and a multi-joint block structure in an image-based virtual try-on model of different sizes.

[0007] Furthermore, clothing size perception includes the following steps:

[0008] Step 1: Automatically assign appropriate size tags to clothing by analyzing the text description of clothing. It identifies descriptive words related to clothing sizes and tags clothing accordingly. The clothing text is preprocessed to ensure accurate keyword extraction. A pre-trained language model is used to identify size-related keywords and map them to size tags 0, 1, 2, and 3. The mapping rules are as follows:

[0009] L s = argmax i ((Softmax(w i ·ε(k))),

[0010] Among them, w i ·ε(k) represents the cumulative weight value of the i-th size label, which is composed of the keyword weight w i It is obtained by weighted summation of the keyword feature ε(k).

[0011] Furthermore, the adaptive mask generation comprises the following steps:

[0012] Step 1: Use human pose estimation to obtain human key points P as the basis for generating masks, and incorporate the previously assigned clothing size labels into the clothing size guidance scale formula. Let S represent the human key point guidance scale based on clothing size labels. The calculation formula is as follows:

[0013] S=α·L s +β

[0014] Among them, α represents the label L s The influence coefficient on size S, β represents the reference size.

[0015] Furthermore, clothing fitting denoising includes the following steps:

[0016] Step 1: Through the adjusted deep learning network, fitting results can be generated for clothing of any size. For denoising, the potential diffusion model is used to convert the original input into a potential feature representation using a variational autoencoder. After sampling and denoising, the potential vector is obtained by the decoder. Decode to generate the target image, use the XL version of the diffusion repair model, and expand the number of channels in the deep learning network to 13 to support denoised input, and connect the following four parts as its input conditions: 1: The noise hidden vector z of ε(x0) t , 2: Adaptive mask m, 3: Hidden vector ε(x a ), 4: Hidden vector ε(x) of dense pose d ): In addition to the above auxiliary condition input, a specific loss function is also used to train Unet, which is expressed as:

[0017]

[0018] Where ζ=Concat[z t ,m,ε(x a ),ε(x d )],ψ=Concat[τ1(x c ),τ2(y)], τ1 and τ2 represent CLIP image encoder and text encoder respectively, ε(x s ) and ε(x s ) represent clothing latent features and clothing sketch latent features respectively.

[0019] Furthermore, clothing learning includes the following steps:

[0020] Step 1: Use a frozen pre-trained deep learning network encoder to capture these features;

[0021] Step 2: Clothing learning uses the deep learning network of the XL version of the stable diffusion inpainting model to enhance the performance of generated images.

[0022] Furthermore, the multi-joint block includes the following steps:

[0023] Step 1: Connect the features in denoising with the intermediate features in frozen clothing learning and input them into the denoising self-attention layer. By retaining the first half of the features and inputting them into the cross-attention layer, the features extracted by the CLIP encoder and the line draft control network are combined. The present invention has the following advantages: The present invention adjusts clothing size through text prompts. The method defines the mask size according to the clothing size information to generate accurate clothing size, enhance user experience, and achieve more realistic and personalized virtual try-on.

[0024] The present invention enhances the control of clothing features by designing multiple joint blocks, by paying more attention to clothing features added to the cross attention during the image generation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a virtual try-on result diagram of different clothing sizes on the VITON-HD data set of the present invention;

[0026] Figure 2 An overview of the image-based virtual fitting model of the present invention;

[0027] Figure 3 is an overview diagram of the multi-joint block of the present invention;

[0028] Figure 4 A comparison chart of the results of the strong baseline of the present invention and the results obtained using different feature strengths during training;

[0029] Figure 5 A supplementary figure of the virtual try-on results of different clothing sizes on the VITON-HD dataset of the present invention;

[0030] Figure 6 A qualitative comparison diagram of the present invention with other methods on the VITON-HD dataset;

[0031] Figure 7 This is a qualitative comparison diagram of the present invention with GP-VTON and Paint-by-Example methods on the Dress Code dataset. DETAILED DESCRIPTION

[0032] The present invention provides a multimodal size-aware virtual try-on method based on a latent diffusion model. The method comprises clothing size perception, adaptive mask generation, clothing try-on denoising, clothing learning and a multi-joint block structure in an image-based virtual try-on model of different sizes.

[0033] In this embodiment, clothing size perception includes the following steps:

[0034] Step 1: Automatically assign appropriate size tags to clothing by analyzing the text description of clothing. It identifies descriptive words related to clothing sizes and tags clothing accordingly. The clothing text is preprocessed to ensure accurate keyword extraction. A pre-trained language model is used to identify size-related keywords and map them to size tags 0, 1, 2, and 3. The mapping rules are as follows:

[0035] L s = argmax i (Softmax(w i ·ε(k))),

[0036] Among them, w i ·ε(k) represents the cumulative weight value of the i-th size label, which is composed of the keyword weight w i It is obtained by weighted summation of the keyword feature ε(k).

[0037] In this embodiment, the adaptive mask generation includes the following steps:

[0038] Step 1: Use human pose estimation to obtain human key points P as the basis for generating masks, and incorporate the previously assigned clothing size labels into the clothing size guidance scale formula. Let S represent the human key point guidance scale based on clothing size labels. The calculation formula is as follows:

[0039] S=α·L s +β

[0040] Among them, α represents the label L s The influence coefficient on size S, β represents the reference size.

[0041] In this embodiment, clothing fitting denoising includes the following steps:

[0042] Step 1: Through the adjusted deep learning network, fitting results can be generated for clothing of any size. For denoising, the potential diffusion model is used to convert the original input into a potential feature representation using a variational autoencoder. After sampling and denoising, the potential vector is obtained by the decoder. Decode to generate the target image, use the XL version of the diffusion repair model, and expand the number of channels in the deep learning network to 13 to support denoised input, and connect the following four parts as its input conditions: 1: The noise hidden vector z of ε(x0) t , 2: Adaptive mask m, 3: Hidden vector ε(x a ), 4: Hidden vector ε(x) of dense pose d ): In addition to the above auxiliary condition input, a specific loss function is used to train Unet, which is expressed as:

[0043]

[0044] Where ζ=Concat[z t ,m,ε(x a ), ε(x d )],ψ=Concat[τ1(x c ), τ2(y)], τ1 and τ2 represent the CLIP image encoder and text encoder respectively, and ε(x s ) represent clothing latent features and clothing sketch latent features respectively.

[0045] In this embodiment, clothing learning includes the following steps:

[0046] Step 1: Use a frozen pre-trained deep learning network encoder to capture these features;

[0047] Step 2: Clothing learning uses the deep learning network of the XL version of the stable diffusion inpainting model to enhance the performance of generated images.

[0048] In this embodiment, the multi-joint block includes the following steps:

[0049] Step 1: Connect the features from denoising with the intermediate features from frozen clothing learning and input them into the denoising self-attention layer. By retaining the first half of the features and inputting them into the cross-attention layer, the features extracted by the CLIP encoder and the line drawing control network are combined.

[0050] In this implementation, Stable Diffusion (SD) is developed and optimized based on Latent Diffusion Model (LDM), which generates images by iteratively applying noise and back-diffusion processes. It combines Variational Autoencoder (VAE), which consists of encoder ε and decoder The model initially projects the input image x into the latent space: z0 = ε(x). During training, the latent features in the model are converted to Gaussian noise using the following function: where t∈{1,...,T} represents the time step of the forward diffusion process. is the number of all α from time step 1 to T t The cumulative product of t Indicates that Gaussian noise is added Given an image x and a text prompt y, the denoising UNet∈ is trained using the following loss formula θ : where τ φ is a CLIP text encoder based on textual prompt y.

[0051] In this embodiment, Figure 1 As shown, it shows the virtual try-on results of different clothing sizes on the VITON-HD dataset. s Indicates the size labels corresponding to different clothing sizes, determined by the user's clothing text prompt.

[0052] In this embodiment, Figure 2 As shown in the figure, given a clothing text prompt, MS-VTON uses clothing size perception and adaptive mask generation modules to predict the clothing size deformation degree s and generate a mask m. Fusion Unet is the main Unet used for training. Mask m, mask image person ε(x a )、Denseposeε(x d ) and noise latent data z t Connected to Fusion Unet as input, as mentioned above, Apparel Unet encodes clothing features, Drafts Controlnet encodes clothing sketch features, and CLIP Encoder combines the encoded clothing text prompt features with clothing features.

[0053] In this embodiment, in the experiment, two public virtual try-on datasets VITON-HD and Dress Code are used for experiments, both of which are high-resolution datasets with a resolution of 1024 × 768. The VITON-HD dataset includes 11,647 pairs of model-clothing pairs for training and 2,032 pairs of model-clothing pairs for testing, and the Dress Code dataset contains more than 53,000 pairs of model-clothing image pairs, which include full-body models and their corresponding upper body clothing, lower body clothing, and dresses, and 1,800 pairs are used for testing in each category.

[0054] In this embodiment, in the experiment, the weights of the Stable Diffusion XL repair model are used to initialize the model. During training, the learning rate is set to 1e-5, and the Adam optimizer with 16-bit mixed precision is used. All results are obtained by training for 10,000 iterations using a dataset with a resolution of 512×384 on a single NVIDIA A100 GPU.

[0055] In this example, the method is compared with three state-of-the-art GAN-based methods (including VITON-HD, HR-VITON and GP-VTON) and three diffusion-based methods LaDI-VTON, DCI-VTON and StableVTION. At the same time, another diffusion-based restoration method and GP-VTON on the Dress Code dataset are also evaluated. To ensure fairness, all experiments in the quantitative evaluation are conducted on images with a resolution of 512×384. LPIPS, SSIM, FID and KID are used as evaluation indicators, and these indicators are achieved following previous work.

[0056] In this example, the multi-joint block is studied in depth on the VITON-HD dataset. First, a virtual try-on task is performed based on a strong baseline, and the images generated under different feature conditions are observed to demonstrate the effectiveness of the multi-joint block in aligning clothing features. Figure 2 As shown, an overview of MS-VTON is presented, where the outlined approach serves as a strong baseline to verify the impact of different feature strengths by reducing the strength of learned clothing features during training. This can be achieved by not using Apparel Unet or Drafts Controlnet, or not using both at the same time.

[0057] In this embodiment, Figure 4As shown in Figure 1, images generated without using clothing features and Drafts-Feature during training exhibit coarse features at the corresponding positions. Images trained with a single clothing feature can better preserve pattern features but inaccurately describe or blur text. On the contrary, using a single Drafts-Feature can align text and pattern positions but cannot fully preserve the semantic features of clothing. The results of the strong baseline clearly restore the text information and accurately preserve the pattern style, confirming that the multi-coupling block effectively restores the original clothing features, as shown in Table 1. The performance indicators also show that the multi-coupling block is highly robust.

[0058] In this embodiment, Figure 6 As shown, sample results of virtual try-on on the VITON-HD dataset are shown, comparing different VTON methods with the present method. After observation, the invented MS-VTON method produces more realistic results while retaining the original clothing features. Among the GAN-based methods, VITON-HD and HR-VITON cannot fully retain the original clothing details and often distort the body. In the second row, GP-VTON has difficulty aligning features on texture pattern clothing. Compared with other LDM-based methods, LaDI-VTON and DCI-VTON change the original clothing features in the first two rows. Stable-VTON and LaDI-VTON also have difficulty in handling complex text or patterns on clothing. In contrast, the present method not only retains most of the clothing features, but also generates images that are closer to the original clothing.

[0059] In this embodiment, Figure 7 As shown in Table 1, a qualitative comparison is made on the Dress Code dataset. As shown in the indicators in Table 1, GP-VTON successfully restores the original image. However, both GP-VTON and Paint-by-Example change the human features; for example, GP-VTON changes the skin color of the person in the first row of results, while Paint-by-Example changes the posture of the person in the second row of results. In contrast, MS-VTON of our method retains the original clothing features without changing the human features, achieving a realistic try-on effect.

[0060] In this embodiment, as shown in Table 2, the quantitative evaluation results of the present method and the state-of-the-art VTON method on the unpaired VITON-HD and Dress Code datasets are presented. The experimental results show that the GAN-based methods (such as GP-VTON) achieve relatively high scores on the SSIM metric, but the scores on LPIPS are lower than the present method. This shows that although GP-VTON can restore the original image, it does not consider high-level semantic features. The present method outperforms the state-of-the-art LDM-based methods in all indicators, does not produce artifacts unrelated to the original clothing, and retains the semantic features of the original clothing.

[0061] Table 1 shows the results of the ablation study on the VITON-HD dataset.

[0062] Table 2 shows a quantitative comparison of the latest methods on the VITON-HD and Dress Code datasets.

[0063] Although the specific implementation of the present invention is described in detail in conjunction with the drawings, it should not be understood as limiting the scope of protection of the present invention. Within the scope described in the claims, various modifications and variations that can be made by those skilled in the art without creative work still fall within the scope of protection of this patent.

[0064] Table 1

[0065]

[0066] Table 2

[0067]

Claims

1. A multimodal size-aware virtual try-on method based on a latent diffusion model, characterized by: The method includes a virtual try-on model of different sizes based on images, which is composed of clothing size perception, adaptive mask generation, clothing try-on denoising, clothing learning and multiple joint blocks.

2. The multimodal size-aware virtual try-on method based on a latent diffusion model as claimed in claim 1, characterized in that: The clothing size perception in claim 1 comprises the following steps: Step 1: Automatically assign appropriate size tags to clothing by analyzing the text description of clothing. It identifies descriptive words related to clothing sizes and tags clothing accordingly. The clothing text is preprocessed to ensure accurate keyword extraction. A pre-trained language model is used to identify size-related keywords and map them to size tags 0, 1, 2, and 3. The mapping rules are as follows: L s =argmax i (Softmax(w i ·ε(k))), Among them, w i ·ε(k) represents the cumulative weight value of the i-th size label, which is composed of the keyword weight w i It is obtained by weighted summation of the keyword feature ε(k).

3. The multimodal size-aware virtual try-on method based on a latent diffusion model as claimed in claim 1, characterized in that: The adaptive mask generation in claim 1 comprises the following steps: Step 1: Use human pose estimation to obtain human key points P as the basis for generating masks, and incorporate the previously assigned clothing size labels into the clothing size guidance scale formula. Let S represent the human key point guidance scale based on clothing size labels. The calculation formula is as follows: S=α·L s +b Among them, α represents the label L s The influence coefficient on size S, β represents the reference size.

4. The multimodal size-aware virtual try-on method based on a latent diffusion model as claimed in claim 1, characterized in that: The clothing fitting denoising in claim 1 comprises the following steps: Step 1: Through the adjusted deep learning network, fitting results can be generated for clothing of any size. For denoising, the potential diffusion model is used to convert the original input into a potential feature representation using a variational autoencoder. After sampling and denoising, the potential vector is obtained by the decoder. Decode to generate the target image, use the XL version of the diffusion repair model, and expand the number of channels in the deep learning network to 13 to support denoised input, and connect the following four parts as its input conditions: 1: The noise hidden vector z of ε(x0) t , 2: Adaptive mask m, 3: Hidden vector ε(x a ), 4: Hidden vector ε(x) of dense pose d ): In addition to the above auxiliary condition input, a specific loss function is also used to train Unet, which is expressed as: Where ζ=Concat[z t ,m,ε(x a ),ε(x d )], τ1 and τ2 represent CLIP image encoder and text encoder respectively, and ε(x s ) represent clothing latent features and clothing sketch latent features respectively.

5. The multimodal size-aware virtual try-on method based on latent diffusion model according to claim 1, characterized in that: The clothing learning in claim 1 comprises the following steps: Step 1: Use a frozen pre-trained deep learning network encoder to capture these features; Step 2: Clothing learning uses the deep learning network of the XL version of the stable diffusion inpainting model to enhance the performance of generated images.

6. The multimodal size-aware virtual try-on method based on a latent diffusion model as claimed in claim 1, characterized in that: The multi-joint block in claim 1 comprises the following steps: Step 1: Connect the features from denoising with the intermediate features from frozen clothing learning and input them into the denoising self-attention layer. By retaining the first half of the features and inputting them into the cross-attention layer, the features extracted by the CLIP encoder and the line drawing control network are combined.

Images