Double-stage potential diffusion system for high-fidelity virtual fitting

Abstract

The invention discloses a two-stage potential diffusion system and method for high-fidelity virtual fitting, and the method comprises the steps: firstly building a semantic corresponding relation between a garment and a human body in a potential space through a first diffusion network, and generating a deformed garment adaptive to a target posture in combination with an input garment image, a human body image and an optional mask image; and then, introducing a cross-modal fusion network based on conditional diffusion, taking the VAE codes of the deformed garment and the original garment image generated in the first stage and the VAE code of the target figure image as input, and finally synthesizing a high-quality virtual fitting image through multi-scale feature fusion and integration of a semantic structure, original garment texture and a human body posture. According to the invention, through a two-stage progressive generation strategy and a specific network structure, challenges of an existing virtual try-on method in the aspects of detail reservation, semantic alignment, training stability, generation quality and the like are solved, and a high-fidelity virtual try-on effect with rich details, vivid textures and matched postures can be generated.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and in particular relates to a two-stage potential diffusion system for high-fidelity virtual try-on. Background Art

[0002] With the rapid development of the e-commerce industry, optimizing the online shopping experience has become a key driver for improving user satisfaction and conversion rates. Virtual Try-On (VTON) technology, a key solution for improving the consumer shopping experience, has garnered widespread attention from both academia and industry in recent years. Users can use VTON to preview how clothing will look on them, thereby assisting with purchasing decisions and reducing returns.

[0003] However, existing virtual try-on methods still face numerous challenges. For example, methods based on generative adversarial networks (GANs) often struggle with training stability and convergence, and are prone to mode collapse or excessive training time. Furthermore, GANs often perform poorly in detail preservation, and the generated images may lack fine clothing textures or exhibit artifacts.

[0004] On the other hand, while some methods based on latent diffusion models (LDMs) have achieved improvements in image generation quality, they still face challenges in virtual try-on tasks, such as preserving clothing details, semantic alignment between the human body and clothing, inference efficiency, and generalization to unseen poses and clothing styles. Specifically, how to accurately "put" a two-dimensional clothing image onto a three-dimensional human model while preserving the clothing's original texture, pattern, and wrinkles, while adapting to different human poses and body shapes, is a complex problem. Furthermore, existing methods still have room for improvement in handling occlusion between clothing and the human body, and in generating natural wrinkles and deformations of clothing on the human body.

[0005] Therefore, developing a high-fidelity virtual try-on technology that can generate rich details, realistic textures, posture matching, and stable training is of great significance to improving the online shopping experience. Summary of the Invention

[0006] In response to the problems raised in the above background technology, the purpose of the present invention is to provide a two-stage latent diffusion system and method for high-fidelity virtual try-on, aiming to improve the fidelity, detail preservation ability and semantic alignment accuracy of virtual try-on images.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A two-stage latent diffusion system for high-fidelity virtual try-on includes: a first-stage processing module, configured to receive a target person image and a clothing image, establish a semantic correspondence between the clothing image and the target person image in a latent space, and generate a deformed clothing image adapted to the posture of the target person image; a second-stage processing module, coupled to the first-stage processing module, configured to receive the deformed clothing image, the clothing image, and the target person image generated by the first-stage processing module, and synthesize and output a final high-fidelity virtual try-on image through cross-modal fusion processing based on conditional diffusion.

[0009] Preferably, the first-stage processing module includes: a first image encoding unit, configured to perform feature extraction on the clothing image, the target person image and an optional mask image to obtain corresponding feature embeddings; a first diffusion network, configured to receive the feature embeddings, gradually denoise in the latent space and establish a semantic correspondence between the clothing image and the target person image, and output a latent representation of the deformed clothing image; and a first image decoding unit, configured to decode the latent representation of the deformed clothing image into the deformed clothing image.

[0010] Preferably, the first diffusion network is a cross-attention warped UNet (CAW-UNet). The CAW-UNet uses a frozen CLIP (Contrastive Language-Image Pre-Training) image encoder and a trainable projection layer to extract texture features of clothing and posture features of the human body. It also uses Q-Former for cross-modal feature projection and fuses the marker embeddings extracted from the person image, clothing image, and mask image through a cross-attention mechanism to ensure accurate alignment of clothing with human posture, thereby generating a preliminary image showing a person wearing clothing, i.e., a deformed clothing.

[0011] Preferably, the second-stage processing module includes: a second image encoding unit, which uses a variational autoencoder (VAE) to encode the clothing image and the target person image respectively to obtain corresponding latent space representations; a second diffusion network, which is a cross-modal fusion network based on conditional diffusion, configured to connect these latent representations along the channel dimension as the main input, and at the same time use the deformed clothing image generated in the first stage as a condition through the frozen CLIP image encoder and the features extracted by the trainable projection layer, and integrate them into the middle layer of the U-Net model through the cross-attention mechanism to perform multi-scale feature fusion processing, effectively integrate semantic structure information, original clothing texture information and human posture information, and output the latent representation of the final high-fidelity virtual try-on image; and a second image decoding unit, which uses the decoder of the VAE to decode the latent representation of the final high-fidelity virtual try-on image into the final high-fidelity virtual try-on image.

[0012] The present invention also provides a corresponding two-stage latent diffusion method for high-fidelity virtual try-on, the steps of which correspond to the functions of the above-mentioned system.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] 1. This invention adopts a two-stage progressive generation strategy. The first stage focuses on the geometric deformation of clothing and its alignment with human posture, while the second stage focuses on the fusion of texture details and high-fidelity rendering. As a result, the final virtual try-on image is more realistic in terms of clothing texture, wrinkles, lighting, etc., and the details are more fully preserved.

[0015] 2. This paper effectively establishes a semantic correspondence between clothing and the human body through the cross-attention and Q-Former mechanisms in CAW-UNet, ensuring the correct wearing form of clothing on the human body and avoiding dislocation and unnatural deformation.

[0016] 3. Compared to traditional GAN ​​methods, the diffusion model-based approach generally has better training stability and generation diversity. The dual-stage design of the present invention further decomposes the task difficulty, helping to improve the overall generation quality and stability.

[0017] 4. The cross-modal fusion network in the second stage of the present invention can effectively integrate multi-source information from the deformed clothing, the original clothing and the target person, ensuring that the generated results are perfectly integrated with the posture and body shape of the target person while maintaining the original characteristics of the clothing.

[0018] 5. The framework proposed in this paper shows good robustness and generalization ability when dealing with complex postures and diverse clothing styles, and can cope with a wider range of virtual try-on scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive work, among which:

[0020] Figure 1 This is a schematic diagram of the structure of the first stage processing module in the present invention;

[0021] Figure 2 Schematic diagram of the structure of the second stage processing module in the present invention;

[0022] Figure 3 Schematic diagram of the process of the two-stage latent diffusion method for high-fidelity virtual try-on of the present invention. DETAILED DESCRIPTION

[0023] The following, in conjunction with specific embodiments, describes in detail the specific steps and operating procedures of a two-stage potential diffusion system for high-fidelity virtual try-on of the present invention, so that ordinary technicians in the relevant technical field can implement the present invention. Through the following specific embodiments, technicians can fully understand and implement the technical solutions of the present invention.

[0024] This paper proposes a two-stage latent diffusion framework named DiffFit for high-fidelity virtual try-on. The method adopts a progressive generation strategy and is divided into two key stages.

[0025] Phase 1: Deformed Clothing Generation

[0026] See also Figure 1 As shown, the goal of the first stage is to generate deformable clothing that adapts to the target person's pose.

[0027] Input: target person image (PersonImage), clothing image (ClothImage), and optional mask image (Mask, such as human segmentation mask or foreground mask).

[0028] Core network: Cross-Attention Warped UNet (CAW-UNet).

[0029] 1. Feature extraction and projection: Use the frozen CLIP (Contrastive Language-Image Pre-Training) image encoder to encode the target person image, clothing image, and mask image respectively to extract their high-level semantic features.

[0030] These features are then passed through a trainable projection layer (Proj) for dimension reshaping and information filtering.

[0031] The Q-Former module is introduced. It is a lightweight Transformer structure that performs cross-modal feature interaction and refinement on projected features. Q-Former can align the features extracted by the cross-attention mechanism with the global input features of the UNet, allowing the model to better integrate and utilize this multimodal information.

[0032] 2. Diffusion process and UNet:

[0033] The noisy image (noisy image refers to the intermediate image that the model gradually denoises during the generation process) is converted to the latent space through a frozen VAE (Variational Autoencoder) encoder.

[0034] Denoising U-Net receives the latent representation of a noisy image as its main input.

[0035] The feature embeddings from the person image, clothing image, and mask image obtained after processing through the CLIP encoder, projection layer, and Q-Former in step 1 are integrated into each layer of the denoising UNet through a cross-attention mechanism. This approach effectively guides the UNet to consider how clothing should deform based on human pose during the denoising process.

[0036] 3. Output:

[0037] The output of the denoising UNet is the latent representation of the deformable clothing.

[0038] The latent representation is then passed through a VAE decoder to reconstruct a distorted clothing image (Distorted Cloth Image / Warped Garment) in pixel space. This distorted clothing image is geometrically adapted to the pose of the target person.

[0039] The loss function L1 of the first stage is defined as:

[0040]

[0041] Among them, z t is the noisy potential representation, ε is the sampled noise, c p ,c c ,c m are the conditional embeddings of person images, clothing images, and mask images, respectively. t is the time step, and εθ It is a denoising network with parameters.

[0042] Phase 2: High-fidelity image synthesis

[0043] See also Figure 2 As shown in Figure 3, the goal of the second stage is to use the deformed clothing generated in the first stage, combined with the original clothing texture and human body information, to synthesize the final high-fidelity virtual try-on result.

[0044] Input: Warped Garment image generated in the first stage, original clothing image (ClothImage), and target person image (Person Image).

[0045] Core network: a cross-modal fusion network based on conditional diffusion (usually also a UNet structure).

[0046] 1. Input encoding and splicing:

[0047] A frozen VAE encoder is used to encode the target person image and the original clothing image respectively, project them into the latent space, and process them together with the latent representation of the noisy image (the intermediate image that the model gradually denoises during the generation process).

[0048] These latent representations are concatenated in the channel dimension and serve as the main input of the subsequent denoising UNet.

[0049] 2. Conditional feature injection:

[0050] The deformed clothing image generated in the first stage also extracts its features through a frozen CLIP image encoder (ImageEncoder) and a trainable projection layer (Projection).

[0051] These features are injected into the middle layer of the second-stage denoising UNet through a cross-attention mechanism. This can use the geometric structure and style information of the deformed clothing as a strong condition to guide the generation of the final image, ensuring the preservation of clothing details and the consistency of style.

[0052] 3. Diffusion and fusion:

[0053] The second stage, the Denoising U-Net, performs the denoising process in the latent space. It fuses the encoded information from the target person, the original clothing, and the conditional features from the deformed clothing in the first stage.

[0054] Through multi-scale feature fusion, the network is able to effectively integrate semantic structure information (from the human body and deformed clothing), original clothing texture (from the original clothing image) and human posture information.

[0055] 4. Output:

[0056] The denoising UNet outputs the latent representation of the final fitting result.

[0057] The latent representation is reconstructed through the VAE decoder to obtain a high-quality, high-fidelity virtual try-on image (Denoised Image / Final Try-on Image).

[0058] The loss function L2 of the second stage is defined as:

[0059]

[0060] Among them, d c represents the conditional embedding of the deformed clothing image output by the first stage, ∈ φ It is the second stage denoising network.

[0061] Specific implementation details of the present invention:

[0062] Datasets: Our method can be trained and evaluated on public virtual try-on datasets such as DressCode, VITON-HD, and IGPair. For example, the Dress Code dataset contains 53,795 high-quality image pairs, and the VITON-HD dataset contains 13,679 image pairs, focusing on complex poses and alignment.

[0063] Training parameters: The AdamW optimizer can be used with a fixed learning rate of 5e-5. Training can be performed on, for example, an NVIDIA RTX 4090 GPU. The batch size can be set to 6, and the training time is approximately 50,000 steps.

[0064] Inference stage: The image generation task can use the UniPC sampler, with the sampling step number set to 50 and the guidance scale parameter w (guidance scale) set to 7.5.

[0065] See also Figure 3 As shown, the overall process of the two-stage latent diffusion method for high-fidelity virtual try-on of the present invention is as follows:

[0066] Step S301: input a target person image, a clothing image, and an optional mask image.

[0067] Step S302, Phase 1: The Phase 1 processing module (e.g., CAW-UNet) performs feature extraction and cross-modal feature fusion on the input image, establishes semantic correspondence between clothing and the human body in the latent space, and uses a diffusion model to generate a deformed clothing image adapted to the target pose. This step corresponds to the detailed description of Phase 1 above.

[0068] Step S303, Phase 2: The deformed garment image, original garment image, and target person image generated in Phase 1 are input into the Phase 2 processing module. The VAE encoder is used to obtain the latent representations of the original garment image and target person image. Combined with the conditional features of the deformed garment image, a cross-modal fusion network based on conditional diffusion (the second diffusion network) is used to perform multi-scale feature fusion and denoising to generate a latent representation of the final fitting result. This step corresponds to the detailed description of Phase 2 above.

[0069] Step S304: Decode the latent representation of the final try-on result through the VAE decoder and output a high-fidelity virtual try-on image.

[0070] The DiffFit framework proposed in this invention significantly outperforms existing virtual try-on methods (such as LaDI-VTON, StableVITON) in both quantitative evaluation (such as LPIPS, SSIM, FID indicators) and qualitative evaluation on multiple datasets (such as VITON-HD). User studies also show that the results generated by this invention are preferred by users in more than 63% of cases. Ablation experiments verify the effectiveness of the two-stage architecture, the necessity of the cross-modal feature encoding module and the clothing condition preservation mechanism, and demonstrate the advantages of this invention in terms of clothing and body alignment accuracy, cross-modal information integration efficiency, and consistency maintenance of clothing details and style.

[0071] In summary, the two-stage latent diffusion system and method proposed in this invention, through a carefully designed network structure and progressive generation strategy, effectively improves the fidelity and realism of virtual try-on images, solves many shortcomings of the existing technology, and has significant technological progress and application value.

[0072] Although specific embodiments of the present invention have been described above, those skilled in the art will appreciate that these specific embodiments are merely illustrative, and that those skilled in the art may omit, substitute, and modify the details of the methods and systems described above without departing from the principles and spirit of the present invention. For example, combining the above method steps to perform substantially the same functions in substantially the same manner to achieve substantially the same results falls within the scope of the present invention. Accordingly, the scope of the present invention is limited solely by the appended claims.

Claims

1. A two-stage latent diffusion system for high-fidelity virtual try-on, characterized by: include: a first-stage processing module configured to receive a target person image and a clothing image, establish a semantic correspondence between the clothing image and the target person image in a latent space, and generate a deformed clothing image adapted to the pose of the target person image; as well as The second-stage processing module is coupled to the first-stage processing module and is configured to receive the deformed clothing image, the clothing image, and the target person image generated by the first-stage processing module, and synthesize and output the final high-fidelity virtual try-on image through cross-modal fusion processing based on conditional diffusion.

2. A two-stage latent diffusion system for high-fidelity virtual try-on according to claim 1, characterized in that: The first stage processing module includes: a first image encoding unit, configured to extract features from the clothing image, the target person image, and an optional mask image to obtain corresponding feature embeddings; a first diffusion network configured to receive the feature embedding, progressively denoise in a latent space and establish a semantic correspondence between the clothing image and the target person image, and output a latent representation of the deformed clothing image; and The first image decoding unit is configured to decode the latent representation of the deformed clothing image into the deformed clothing image.

3. A two-stage latent diffusion system for high-fidelity virtual try-on according to claim 2, characterized in that: The first diffusion network is the Cross Attention Warped UNet (CAW-UNet); The CAW-UNet consists of a frozen CLIP image encoder and a trainable projection layer for extracting and projecting features of each input image; The CAW-UNet also includes a Q-Former module for cross-modal feature projection; The CAW-UNet fuses the marker embeddings extracted from the target person image, the clothing image, and the mask image through a cross-attention mechanism to ensure accurate alignment of the deformed clothing image with the target person image pose.

4. A two-stage latent diffusion system for high-fidelity virtual try-on according to claim 1, characterized in that: The second stage processing module includes: a second image encoding unit, encoding the clothing image and the target person image using a variational autoencoder (VAE) to obtain corresponding latent space representations; a second diffusion network, which is a cross-modal fusion network based on conditional diffusion, configured to receive the concatenated latent space representation of the clothing image and the latent space representation of the target person image as primary input, integrate semantic structure information, original clothing texture information, and human posture information through multi-scale feature fusion processing, and output the latent representation of the final high-fidelity virtual try-on image; and The second image decoding unit uses the decoder of the VAE to decode the latent representation of the final high-fidelity virtual try-on image into the final high-fidelity virtual try-on image.

5. A two-stage latent diffusion system for high-fidelity virtual try-on according to claim 4, characterized in that: When performing multi-scale feature fusion processing, the second diffusion network also uses features extracted from the deformed clothing image generated in the first stage as conditional input to influence the intermediate layers of the second diffusion network and control the overall style of the final high-fidelity virtual try-on image.

6. A two-stage latent diffusion system for high-fidelity virtual try-on according to claim 5, characterized in that: The process of extracting features from the deformed clothing images generated by the first stage includes: extracting features using a frozen CLIP image encoder and a trainable projection layer, and incorporating the features into the second diffusion network through a cross-attention mechanism.

7. A two-stage latent diffusion method for high-fidelity virtual try-on, characterized by: The following steps are involved: The first stage processing steps are: receiving a target person image and a clothing image, establishing a semantic correspondence between the clothing image and the target person image in a latent space, and generating a deformed clothing image adapted to the pose of the target person image; as well as The second stage processing step: receiving the deformed clothing image, the clothing image and the target person image generated by the first stage processing step, synthesizing and outputting the final high-fidelity virtual try-on image through cross-modal fusion processing based on conditional diffusion.

8. A two-stage latent diffusion method for high-fidelity virtual try-on according to claim 7, characterized in that: The first stage processing steps include: Performing feature extraction on the clothing image, the target person image, and an optional mask image to obtain corresponding feature embeddings; The method includes stepwise denoising in a latent space and establishing a semantic correspondence between the clothing image and the target person image through a first diffusion network, outputting a latent representation of the deformed clothing image, and decoding the latent representation of the deformed clothing image into the deformed clothing image.

9. A two-stage latent diffusion method for high-fidelity virtual try-on according to claim 8, characterized in that: The first diffusion network is a Cross-Attention Warped UNet (CAW-UNet), which fuses the marker embeddings extracted from the target person image, the clothing image and the mask image through a cross-attention mechanism to ensure accurate alignment of the deformed clothing image with the target person image posture.

10. The two-stage latent diffusion method for high-fidelity virtual try-on according to claim 7, characterized in that: The second stage processing steps include: Using a variational autoencoder (VAE) to encode the clothing image and the target person image to obtain corresponding latent space representations; A cross-modal fusion network based on conditional diffusion receives the concatenated latent space representation of the clothing image and the latent space representation of the target person image as the main input, injects the features extracted by the image encoder and projection layer of the deformed clothing image generated in the first stage through a cross-attention mechanism, performs multi-scale feature fusion processing, integrates semantic structure information, original clothing texture information, and human posture information, and outputs the latent representation of the final high-fidelity virtual try-on image; and The decoder of the VAE is used to decode the latent representation of the final high-fidelity virtual try-on image into the final high-fidelity virtual try-on image.

Images