Human-to-human virtual fitting method in real scene based on non-training diffusion model
By extracting deep features using a training-free diffusion model and introducing high-frequency feature loss, the problem of low fidelity of clothing texture details in real-world scenarios in existing virtual try-on methods is solved, thus achieving high-quality human-to-human virtual try-on.
Patent Information
- Application Number
- CN202510972599.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-15
- Publication Date
- 2025-10-31
AI Technical Summary
Existing virtual fitting methods rely on large-scale training data and require complex input conditions, making them difficult to adapt to real-world person-to-person fitting scenarios, and they also have low fidelity in clothing texture details.
A training-free diffusion model is adopted, and deep features are extracted through a pre-trained diffusion model to establish semantic correspondence. Combined with deformation, filling, appearance transfer and high-frequency feature loss, human-to-human virtual try-on in real-world scenarios is realized.
No additional training is required; users only need to provide a clothing area mask to achieve high-quality person-to-person virtual try-on in real-world scenarios, preserving clothing details and maintaining background consistency.
Smart Images
Figure QLYQS_1 
Figure QLYQS_3 
Figure QLYQS_4
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of virtual fitting and computer vision technology, and particularly relates to a human-to-human virtual fitting method based on a training-free diffusion model in real-world scenarios. The aim is to achieve high-quality human-to-human virtual fitting in real-world scenarios with minimal input and no additional training required, while improving the fidelity of clothing texture details. Background Technology
[0002] With the rapid development of the e-commerce industry, providing users with convenient and personalized virtual try-on services has become an important demand. The goal of virtual try-on is to render specific clothing onto a given image of a person, generating a realistic and natural try-on effect. This technology can enhance the user shopping experience and narrow the gap between online and offline retail.
[0003] Early virtual try-on methods were primarily based on Generative Adversarial Networks (GANs). These methods typically rely on large-scale datasets and prior knowledge of human anatomy (such as DensePose) to train a deformable network that aligns with clothing features, and then use a trainable generator to produce the final try-on result. Although capable of handling simple virtual try-on tasks, the instability of GANs often leads to poor image quality, especially loss of clothing details, and the dependence on complex inputs such as DensePose increases the burden on users, making them unsuitable for practical applications.
[0004] In recent years, diffusion models have achieved significant success in image generation, and an increasing number of virtual try-on tasks are adopting diffusion-based methods. Early diffusion model methods, similar to GANs, used trainable deformable and generative modules conditioned on human pose or structural features. These methods rely on the alignment capabilities of the deformable modules and exhibit poor generalization. Recent methods employ a reference network-based strategy, training a separate network to encode clothing and injecting its features into the main generative network. While achieving high-quality image synthesis, this still requires substantial computational resources for training and relies on redundant human structural inputs, limiting its practicality in real-world scenarios. Furthermore, existing methods primarily focus on studio scenarios, where clothing is presented as a product image without a background, and the target person's image is captured in a controlled environment. However, in many cases, users expect virtual try-on to handle real-world photographs taken in unconstrained environments, where reference clothing may appear on someone else rather than in a clean product image. Directly applying existing methods to real-world person-to-person try-on often results in background artifacts and structural inconsistencies. While some general-purpose techniques can adapt to this task and maintain background consistency, they often fail to preserve clothing texture details, making them unsuitable for practical virtual try-on applications.
[0005] In summary, current virtual try-on methods suffer from several drawbacks, including reliance on large-scale training data, complex input requirements, difficulty adapting to real-world person-to-person try-on scenarios, and low fidelity in clothing texture details. Therefore, a method is needed that requires no additional training, has low input requirements, enables high-quality person-to-person virtual try-on in real-world scenarios, and preserves clothing details. Summary of the Invention
[0006] To address the above problems, this invention proposes a virtual try-on method based on a training-free diffusion model, which can achieve high-quality person-to-person virtual try-on in real-world scenarios. The method specifically includes the following steps:
[0007] 1. A method for person-to-person virtual try-on in real-world scenarios based on a training-free diffusion model, characterized by the fact that no additional training is required; only a mask indicating the clothing area is provided by the user. A pre-trained diffusion model can then be used to extract deep features and establish semantic correspondences. Through steps such as deformation, filling, appearance transfer, and high-frequency feature loss guidance, person-to-person virtual try-on in real-world scenarios is achieved. The method includes the following steps:
[0008] S1, Constructing semantic correspondence: Using the intermediate layer of the pre-trained diffusion model UNet, deep semantic features are extracted from the clothing region of the target person image and clothing image, and pixel-level semantic correspondence is established through similarity matching;
[0009] S2, Clothing Deformation and Initial Fitting Result Generation: Based on the semantic correspondence obtained in S1, the non-rigid deformation method (MLS) based on control points is used to align the clothing and naturally fit it onto the target person. Then, the pre-trained StableDiffusion-Inpainting model is used in conjunction with the clothing text description to fill the uncovered areas and generate the initial fitting result.
[0010] S3, Appearance Transfer and Detail Optimization: By using DDIM inversion, the clothing image and the initial fitting results are transformed into the latent space, and a mask-appearance attention (MAA) is constructed. This allows the clothing area to query features from the reference image, while the background area retains the original input features, thus achieving appearance transfer to enhance clothing details and maintain background consistency.
[0011] S4, Introduce high-frequency feature loss: Calculate the high-frequency difference between the generated garment and the reference garment, inject the gradient into the noise map to guide the denoising process, improve the fidelity of the garment texture details, and obtain the final fitting image.
[0012] 2. The method for real-world person-to-person virtual try-on based on a training-free diffusion model as described in claim 1, characterized in that the construction of semantic correspondence in step S1 includes the following steps:
[0013] S11, Use SAM to acquire the target person image Xp And clothing images X g Corresponding clothing area mask;
[0014] M p =SAM(X) p ),M g =SAM(X) g (1.1)
[0015] Where M p M g These are the masks for the clothing areas of the person and clothing images, respectively.
[0016] S12, using the intermediate layer of the pre-trained diffusion model UNet as a feature extractor, extracts deep semantic features from the clothing region of the target person image and clothing image, as expressed in the following expression;
[0017] F p =φ(X p M p );F g =φ(X g M g (1.2)
[0018] Where φ represents the intermediate layer of UNet, and F p ,F g These represent the diffusion features of the target image and the clothing image, respectively.
[0019] S13, calculate the similarity-based mapping matrix S between the two features to establish a semantic correspondence, expressed as follows;
[0020] S ij =sim(f p i ,f g i (1.3)
[0021] Among them, f p i ,f g i ∈F p ,F g represents the feature vectors at the i-th and j-th positions in the person image and clothing image, respectively; sim represents the similarity calculation.
[0022] S14, Based on the corresponding matrix, establish semantic control point pairs; for the i-th position in the clothing, search for the most similar position j in the person image. * This part can be represented as:
[0023] j * =argmax j Sij (1.4)
[0024] Using the above formula, we can obtain a pair of control points. g i , These are points on the clothing image and the target person image, respectively. After the above operations, a total of N pairs of control points are obtained.
[0025] 3. The method for person-to-person virtual try-on in a real-world scenario based on a training-free diffusion model as described in claim 1, characterized in that the clothing deformation and initial result generation in step S2 includes the following steps:
[0026] S21, N pairs of control points obtained based on S14 The Moving Least Squares (MLS) method is used for deformation, for any pixel x∈X in the clothing image. g The deformation position f(x) can be expressed as:
[0027] f(x)=R(xg * )+p * (1.5)
[0028] Where g * ,p * It is the weighted center position, and the calculation process can be expressed as:
[0029]
[0030] Where ω i These are the weights of the control points, and the calculation process can be expressed as:
[0031]
[0032] Where ε is a random term. R is a locally rigid rotation matrix, which can be solved by the following optimization problem:
[0033]
[0034] This optimization problem can be solved using Singular Value Decomposition (SVD); by applying the deformation function f(x) to all pixel coordinates, the clothing is deformed and seamlessly composited onto the image of the person, resulting in the output X. * .
[0035] S22, using Stable Diffusion-inpainting to fill uncovered areas, the expression is:
[0036] X init =Inpainting(X *M d C g (1.9)
[0037] Among them, X init The initial fitting image. M d C is the mask for the uncovered area. g The text description of the clothing is in the format: one {sleeve length}{collar shape}{clothing type}.
[0038] 4. The method for person-to-person virtual try-on in a real-world scenario based on a training-free diffusion model as described in claim 1, characterized in that step S3, appearance transfer and detail optimization, specifically comprises:
[0039] S31, the clothing image and the initialization image are converted to the noise space using DDIM·Inversion. This part can be represented as:
[0040]
[0041] in A noise map representing the latent space.
[0042] S32, will Linear projection is used to compute the query Q, key K, and value V, which can be represented as:
[0043]
[0044] Among them W Q W K W V This represents a linear projection matrix.
[0045] S33, followed by mask-appearance attention calculation, for Calculate standard self-attention:
[0046]
[0047] in This represents the result of attention calculation; for Clothing area from Attention calculations are performed to obtain appearance information. Attention calculations are still performed on non-clothing areas relative to themselves; this part can be represented as:
[0048]
[0049] Among them, M g -1 M i -1 They represent 1-M respectively g ,1-M i ;
[0050] S34, the final attention calculation result of mask-appearance attention is expressed as:
[0051]
[0052] Where ⊙ represents pixel-by-pixel multiplication. Then... and Latent variables used for poor attention calculations to obtain the next time step and
[0053] 5. The method for person-to-person virtual try-on in a real-world scenario based on a training-free diffusion model as described in claim 1, characterized in that the high-frequency feature loss introduced in step S4 specifically comprises:
[0054] S41, obtain the final image predicted at each time step; the forward noise addition process at each time step is represented as:
[0055]
[0056] Where ε∈N(0,1), replace ε with the prediction noise ε t This yields the final image predicted at each time step. This part can be represented as:
[0057]
[0058] S42, will predict The image X is obtained by converting it to pixel space using a decoder. i ;
[0059] S43, Calculate X within the clothing area i and X g The frequency domain distance loss can be expressed as:
[0060]
[0061] Where F represents the two-dimensional discrete Fourier transform (DFT), L f This represents the calculated frequency domain distance loss;
[0062] S43, Calculate the gradient of the loss and update This part can be represented as:
[0063]
[0064] Where λ is a weighting factor controlling the guiding intensity; after multiple denoising steps, the final fitting image X is obtained. final .
[0065] Beneficial Effects: Compared with existing technologies, the present invention provides a real-world virtual try-on method based on a training-free diffusion model, enabling high-quality real-world virtual try-on and producing the following beneficial effects:
[0066] 1. No additional training required and low input requirements: This invention uses a pre-trained diffusion model, which requires no additional training. Users only need to provide a mask indicating the clothing area, which greatly improves usability and lowers the barrier to entry for users.
[0067] 2. Adapting to real-world virtual try-on: This invention can process images with any real-world background, enabling virtual try-on in real-world scenarios. This overcomes the limitation of existing methods to studio settings and better meets practical needs.
[0068] 3. Improve the fidelity of clothing texture details: By introducing high-frequency feature loss, the diffusion model is guided to focus on high-frequency features during the denoising process, effectively preserving details such as clothing texture and edges, thereby improving the quality of virtual try-on and user experience.
[0069] 4. Maintain consistency between background and non-clothing areas: Employ an appearance transfer strategy to enhance clothing details while preserving the characteristics of the background and non-clothing areas, avoiding the loss of attributes in non-clothing areas and generating more realistic try-on results. Attached Figure Description
[0070] Figure 1 This is a visualization of the results of the present invention on two virtual try-on tasks.
[0071] Figure 2 This is a flowchart illustrating how the present invention enables virtual try-on for individuals in a cognitive setting.
[0072] Figure 3 This is a visualization comparison of the results of this invention with other methods.
[0073] Figure 4 This is a visualization of the results of the present invention on a traditional virtual try-on task.
[0074] Figure 5 This is a quantitative comparison chart of the present invention with existing methods on the Street-Tryon dataset.
[0075] Figure 6 This is a quantitative comparison chart of the present invention with existing methods on the VTON-HD and Dresscode datasets.
[0076] Figure 7 This is a comparison diagram of the input conditions of the present invention with those of existing methods. Detailed Implementation
[0077] The present invention provides a training-free human-to-human virtual try-on method based on a pre-trained diffusion model. This method uses a pre-trained diffusion model to extract semantic correspondences and achieves human-to-human virtual try-on in real-world scenarios through steps such as deformation, filling, appearance transfer, and high-frequency feature loss guidance. It requires no additional training, only minimal input, and ensures the consistency of clothing details and background.
[0078] The invention will be further described below with reference to specific embodiments.
[0079] Example 1:
[0080] like Figure 2 As shown, the person-to-person virtual try-on process is as follows: The user inputs an image of the person wearing the target clothing (clothing image) and an image of the person to be tried on (target image), as well as a mask indicating the clothing area. First, using the intermediate layer of the pre-trained diffusion model's UNet, deep semantic features are extracted from the clothing areas of the target image and the clothing image. Pixel-level semantic correspondences are established through similarity matching to obtain control point pairs.
[0081] Based on control point pairs, the MLS method is used to perform non-rigid deformation of the clothing, aligning it and naturally fitting it onto the target person to obtain a preliminary composite result. Then, the Stable Diffusion-Inpainting model is used, combined with the text description of the clothing (such as "a short-sleeved U-neck dress"), to fill the uncovered areas and generate an initial fitting result.
[0082] In the refinement stage, the clothing image and initial fitting results are transformed into a latent space using DDIM inversion, and a MAA (Appearance Assignment Analysis) is constructed. This allows clothing areas to retrieve features from the clothing image, while non-clothing areas retain features from the initial results, achieving appearance transfer. Simultaneously, a high-frequency feature loss is introduced to calculate the high-frequency differences between the generated clothing and the reference clothing, and gradients are injected into the noise map to guide the denoising process. After multiple inference steps, the final fitting image is obtained.
[0083] The human-to-human virtual try-on method based on a pre-trained diffusion model without a training framework provided by this invention can generate high-quality human-to-human virtual try-on results, adapt to the needs of real-world scenarios, and support virtual try-on tasks in traditional studio scenarios, with good generalization ability.
[0084] Figure 1 The images show the visualization results of this invention on two virtual try-on tasks. It can be seen that this invention has good generalization ability, achieving ideal try-on results in both tasks.
[0085] Figure 2This is a flowchart illustrating the process of implementing virtual try-on in a human-to-human context according to the present invention. It can be seen that the present invention uses only a pre-trained diffusion model as the base model, requiring no additional training or fine-tuning. Furthermore, the task can be completed using only two image masks, eliminating the need for redundant prior knowledge of human features, thus significantly reducing the difficulty of operation for users.
[0086] Figure 3 This is a visualization comparison of the results of this invention with other methods. It is evident that this invention is far superior to other methods in maintaining the accuracy of ancient costume textures. Furthermore, this invention is better able to preserve the consistency of non-clothing areas, such as arm tattoos and watches.
[0087] Figure 4 This is a visualization of the results of this invention on a traditional virtual try-on task. It can be seen that, for traditional tasks, this invention achieves competitive results in terms of texture consistency and overall quality.
[0088] Figure 5 This is a quantitative comparison chart of the present invention with existing methods on the Street-Tryon dataset. It can be seen that the present invention outperforms existing methods in texture consistency (CLIP-I and DINO-I), image quality (FID and FID), consistency of non-clothing areas, and hand consistency.
[0089] Figure 6 This chart shows a quantitative comparison between the present invention and existing methods on the VTON-HD and Dresscode datasets. It is evident that the present invention achieves the best results among training-free methods. It also achieves competitive results among training-based methods.
[0090] Figure 7 This diagram compares the input conditions of the present invention with those of existing methods. It is evident that the present invention does not require additional conditions such as Densepose or agnostic maps; a mask is sufficient to achieve high-quality virtual try-on.
[0091] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
[0092] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.
Claims
1. A method for person-to-person virtual try-on in real-world scenarios based on a training-free diffusion model, characterized in that, No additional training is required; users only need to provide a mask indicating the clothing area. A pre-trained diffusion model can then be used to extract deep features and establish semantic correspondences. Through steps such as deformation, filling, appearance transfer, and high-frequency feature loss guidance, person-to-person virtual try-on in real-world scenarios can be achieved, including the following steps: S1, Constructing semantic correspondence: Using the intermediate layer of the pre-trained diffusion model UNet, deep semantic features are extracted from the clothing region of the target person image and clothing image, and pixel-level semantic correspondence is established through similarity matching; S2, Clothing Deformation and Initial Fitting Result Generation: Based on the semantic correspondence obtained in S1, the non-rigid deformation method (MLS) based on control points is used to align the clothing and naturally fit it onto the target person. Then, the pre-trained StableDiffusion-Inpainting model is used in conjunction with the clothing text description to fill the uncovered areas and generate the initial fitting result. S3, Appearance Transfer and Detail Optimization: The clothing image and initial fitting results are transformed into the latent space through DDIM inversion, and a mask-appearance attention (MAA) is constructed so that the clothing area queries features from the reference image, while the background area retains the original input features, thereby achieving appearance transfer to enhance clothing details and maintain background consistency. S4, Introduce high-frequency feature loss: Calculate the high-frequency difference between the generated garment and the reference garment, inject the gradient into the noise map to guide the denoising process, improve the fidelity of the garment texture details, and obtain the final try-on image.
2. The method for person-to-person virtual try-on in a real-world scenario based on a training-free diffusion model as described in claim 1, characterized in that, The construction of semantic correspondence in step S1 includes the following steps: S11, Use SAM to acquire the target person image X p And clothing images X g Corresponding clothing area mask; M p =SAM(X p ),M g =SAM(X g ) (1.1) Where M p M g These are the masks for the clothing areas of the person and clothing images, respectively. S12, using the intermediate layer of the pre-trained diffusion model UNet as a feature extractor, extracts deep semantic features from the clothing region of the target person image and clothing image, as expressed in the following expression; F p =φ(X p ,M p );F g =φ(X g ,M g ) (1.2) Where φ represents the intermediate layer of UNet, and F p ,F g These represent the diffusion features of the target image and the clothing image, respectively. S13, calculate the similarity-based mapping matrix S between the two features to establish a semantic correspondence, expressed as follows; S ij =sim(f p i ,f g i ) (1.3) Among them, f p i ,f g i ∈F p ,F g represents the feature vectors at the i-th and j-th positions in the person image and clothing image, respectively; sim represents the similarity calculation. S14, Based on the corresponding matrix, establish semantic control point pairs; for the i-th position in the clothing, search for the most similar position j in the person image. * This part can be represented as: j * =argmax j S ij (1.4) Using the above formula, we can obtain a pair of control points. g i , These are points on the clothing image and the target person image, respectively. After the above operations, a total of N pairs of control points are obtained.
3. The method for person-to-person virtual try-on in a real-world scenario based on a training-free diffusion model as described in claim 1, characterized in that... The clothing deformation and initial result generation in step S2 include the following steps: S21, N pairs of control points obtained based on S14 The Moving Least Squares (MLS) method is used for deformation, for any pixel x∈X in the clothing image. g The deformation position f(x) can be expressed as: f(x)=R(x-g * )+p * (1.5) Where g * ,p * It is the weighted center position, and the calculation process can be expressed as: Where ω i These are the weights of the control points, and the calculation process can be expressed as: Where ε is a random term. R is a locally rigid rotation matrix, which can be solved by the following optimization problem: This optimization problem can be solved using Singular Value Decomposition (SVD); by applying the deformation function f(x) to all pixel coordinates, the clothing is deformed and seamlessly composited onto the image of the person, resulting in the output X. * . S22, using Stable Diffusion-inpainting to fill uncovered areas, the expression is: X init =Inpainting(X * ,M d ,C g )(1.9) Among them, X init Initialized try-on image. M d C is the mask for the uncovered area. g The text description of the clothing is in the format: one {sleeve length}{collar shape}{clothing type}.
4. The method for person-to-person virtual try-on in a real-world scenario based on a training-free diffusion model as described in claim 1, characterized in that... The specific steps of S3, appearance migration and detail optimization, are as follows: S31, convert the clothing image and the initialization image to a noisy space using DDIM·Inversion.