Virtual fitting method and system based on bootstrap end-to-end diffusion model

Abstract

The invention discloses a virtual fitting method and system based on a bootstrap end-to-end diffusion model, and the method comprises the steps: carrying out the training of generating a synthetic image based on a figure image and random clothes, and constructing an initial pseudo triple data set; inputting the initial pseudo triple data set into an end-to-end diffusion model for training to obtain a trained end-to-end diffusion model; generating a new pseudo-triple data set based on the trained end-to-end diffusion model, and performing iterative training on the end-to-end diffusion model until the performance is saturated to obtain a bootstrap end-to-end diffusion model; and designing a weight initialization path and a series of pseudo triple generation modes, and training a bootstrap end-to-end diffusion model to realize virtual fitting of multiple scenes. According to the method, a path can be initialized through a specific model weight, and knowledge migration from a simple task to a complex task is realized. The virtual fitting method and system based on the bootstrap end-to-end diffusion model can be widely applied to the technical field of computer vision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a virtual try-on method and system based on a bootstrap end-to-end diffusion model. BACKGROUND

[0002] With the rapid development of e-commerce and online fashion industry, it has become a major pain point that consumers cannot try on clothes in person. Image-based virtual try-on (VTO) technology has emerged as the times require. Current VTO tasks can be mainly divided into four categories of scenarios, with difficulty increasing in turn: store-to-model (G2M), store-to-street (G2S), model-to-model (M2M) and street-to-street (S2S). Most existing researches mainly focus on the most basic G2M task, and are usually designed for a specific scenario, lacking a unified framework to solve all four tasks. At the same time, existing researches usually adopt a two-stage processing flow, leading to the problem of error accumulation, and the generated images are often accompanied by artifacts.

[0003] The mainstream VTO methods (such as CatVTON, StableVITON, GP-VTON, etc.) in the related art usually adopt a two-stage processing flow. The first stage is a preprocessing stage, which generates a semantic segmentation map using a human parsing model, and then generates a mask using the semantic segmentation map, to mask the person image to obtain a masked person image. Some methods also need to extract skeleton points using a pose estimation model, and then "warp" the target clothes through thin-plate spline interpolation (TPS) or flow network (Flow) to align to the body area of the target person. The second stage is a generation stage, which inputs the clothes or the warped clothes, the masked person image into a generation network (such as U-Net or GAN), and synthesizes the final try-on image using image inpainting technology.

[0004] However, the two-stage method highly depends on the accuracy of preprocessing. If human parsing or clothes warping has errors (for example, the mask is not accurate, and the clothes are over-distorted), these errors will be directly passed to the generation stage, resulting in artifacts, damage or unnatural seams in the final image. In addition, it is usually designed for a single task (mainly simple G2M), and it is difficult to directly generalize to scenarios with complex backgrounds or variable poses (such as S2S). SUMMARY

[0005] In order to solve the above technical problems, the purpose of the present application is to provide a virtual try-on method and system based on a bootstrap end-to-end diffusion model, which can realize knowledge transfer from simple tasks to complex tasks through a specific model weight initialization path.

[0006] The first technical solution adopted in this invention is: a virtual try-on method based on a bootstrap end-to-end diffusion model, comprising the following steps: Based on human images and random clothing, we train the generated synthetic images to construct an initial pseudo triplet dataset; The initial pseudo triplet dataset is input into the end-to-end diffusion model for training, resulting in the trained end-to-end diffusion model. A new pseudo triplet dataset is generated based on the trained end-to-end diffusion model. The end-to-end diffusion model is iteratively trained until the model performance saturates, resulting in a bootstrap end-to-end diffusion model. Design a weight initialization path and a series of pseudo triple generation methods, and train a bootstrap end-to-end diffusion model to achieve virtual try-on in multiple scenarios.

[0007] Furthermore, the step of generating synthetic images based on person images and random clothing to construct an initial pseudo-triple dataset specifically includes: Select a person image, a first clothing image, and a second clothing image. Input the person image and the second clothing image into the VTO generator to composite them and obtain a composite image. Training triplets are constructed by combining the synthesized image, the first clothing image, and the character image. The synthesis steps are repeated to construct an initial pseudo-triplet dataset.

[0008] Furthermore, the step of inputting the initial pseudo-triple dataset into the end-to-end diffusion model for training to obtain the trained end-to-end diffusion model specifically includes: The initial pseudo triplet dataset is input into the end-to-end diffusion model, which includes an encoder, a denoising network, and a decoder. The encoder based on the end-to-end diffusion model encodes the initial pseudo triplet dataset into latent variables, obtaining latent variables of people, clothing, and fitting targets. Randomly select a time step and sample noise, and add the noise to the latent variable of the trial target based on the time step to obtain the noisy trial target latent variable; A denoising network based on an end-to-end diffusion model is used to predict noise by taking input noisy latent variables of the target, time step, person, and clothing. The L2 loss of predicted noise and real noise is calculated, and the parameters of the denoising network are optimized by gradient descent and backpropagation algorithms to obtain the trained end-to-end diffusion model.

[0009] Furthermore, the step of generating a new pseudo-triple dataset based on the trained end-to-end diffusion model, and iteratively training the end-to-end diffusion model until its performance saturates to obtain a bootstrap end-to-end diffusion model, specifically includes: Based on the trained end-to-end diffusion model, the human image and random clothing are synthesized to obtain a new synthetic image. A new pseudo triplet dataset is constructed based on the new synthetic images, and the trained end-to-end diffusion model is fine-tuned to obtain a new end-to-end diffusion model. The process of iterating through new image synthesis steps and fine-tuning the trained end-to-end diffusion model continues until the model performance saturates, resulting in a bootstrap end-to-end diffusion model.

[0010] Furthermore, the step of synthesizing a new synthetic image by combining a person image with random clothing based on the trained end-to-end diffusion model specifically includes: Input the image of the person and random clothing into the trained end-to-end diffusion model; The encoder based on the trained end-to-end diffusion model encodes the human image and random clothing to obtain the latent variables of the human image and the latent variables of the random clothing. A denoising network based on a trained end-to-end diffusion model randomly samples noise and concatenates the latent variables of the person image with random clothing latent variables to denoise, resulting in a denoised concatenated image. The decoder based on the trained end-to-end diffusion model decodes the denoised stitched image to obtain a new synthetic image.

[0011] Furthermore, the step of designing a weight initialization path and a series of pseudo-triple generation methods, and training a bootstrap end-to-end diffusion model to achieve virtual try-on in multiple scenarios, specifically includes: We obtained a standard studio shooting dataset and performed iterative bootstrapping training on the end-to-end diffusion model to obtain an end-to-end diffusion model that realizes the store-to-model task. The end-to-end diffusion model for realizing the store-to-model task is initialized and trained iteratively by bootstrapping. The background of the person image and the real image is replaced with the street scene background to obtain the end-to-end diffusion model for realizing the store-to-street photography task. The end-to-end diffusion model for realizing the store-to-model task is initialized and trained iteratively by bootstrapping to generate pseudo triples, thus obtaining the end-to-end diffusion model for realizing the model-to-model task. The end-to-end diffusion model for realizing the store-to-model task is initialized and trained iteratively by bootstrapping. The backgrounds of the person image and the real image are replaced with the same street scene background, and the backgrounds of the conditional clothing image are replaced with different street scene backgrounds to obtain the end-to-end diffusion model for realizing the street photography to street photography task.

[0012] The second technical solution adopted in this invention is: a virtual try-on system based on a bootstrap end-to-end diffusion model, comprising: The first module is used to train the generated synthetic images based on human images and random clothing, and to build an initial pseudo triplet dataset. The second module is used to input the initial pseudo triplet dataset into the end-to-end diffusion model for training, and obtain the trained end-to-end diffusion model. The third module is used to generate a new pseudo triplet dataset based on the trained end-to-end diffusion model, and to iteratively train the end-to-end diffusion model until the model performance is saturated, thus obtaining a bootstrap end-to-end diffusion model. The fourth module is used to design a weight initialization path and a series of pseudo triple generation methods, and to train a bootstrap end-to-end diffusion model to realize virtual try-on in multiple scenarios.

[0013] The beneficial effects of the method and system of this invention are as follows: This invention generates synthetic images based on human images and random clothing to construct an initial pseudo-triple dataset. This initial pseudo-triple dataset is then input into an end-to-end diffusion model for training, resulting in a trained end-to-end diffusion model. Through a bootstrap end-to-end training method, the model is iteratively refined by generating higher-quality pseudo-triples, enabling it to generate images with more realistic details and fewer artifacts. Furthermore, a new pseudo-triple dataset is generated based on the trained end-to-end diffusion model, and the end-to-end diffusion model is iteratively trained until its performance saturates, resulting in a bootstrap end-to-end diffusion model. Finally, a weight initialization path and a series of pseudo-triple generation methods are designed, and the bootstrap end-to-end diffusion model is trained to achieve virtual try-on in multiple scenarios. Utilizing the difficulty hierarchy among four tasks, and through a specific model weight initialization path, knowledge transfer from simple to complex tasks is achieved. Attached Figure Description

[0014] Figure 1 This is a flowchart of the steps of a virtual try-on method based on a bootstrap end-to-end diffusion model according to the present invention. Figure 2 This is a structural block diagram of a virtual try-on system based on a bootstrap end-to-end diffusion model according to the present invention. Figure 3 This is a schematic diagram of the bootstrap iterative training process provided in a specific embodiment of the present invention; Figure 4 This is a schematic diagram of the course learning process provided in a specific embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the differences in synthesized images provided in a specific embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the cumulative error of the results provided in a specific embodiment of the present invention; Figure 7 This is a schematic diagram of high-quality M2M fitting results provided by a specific embodiment of the present invention; Figure 8 This is a schematic diagram of high-quality S2S fitting results provided by a specific embodiment of the present invention. Detailed Implementation

[0015] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0016] First, the technical terms used in the embodiments of this invention will be explained: 1) Virtual Try-On (VTO): This refers to the technology of using computer vision and graphics to composite selected clothing images onto a person's image, making the person appear as if they are wearing the clothing.

[0017] 2) End-to-End (E2E): This refers to a mode that takes raw data (such as images of people and clothing) as input and outputs the final result (fitting images) directly, without going through independent step-by-step processing such as human body analysis and clothing deformation in between.

[0018] 3) Bootstrapping: An iterative optimization strategy that uses the output generated by the model itself as pseudo-data to train the model, thereby gradually improving the model performance.

[0019] 4) Curriculum Learning (CrL): A training strategy that simulates the human learning process, allowing the model to first learn simple tasks and then use the learned knowledge to learn more complex tasks.

[0020] 5) Pseudo Triplets: Data pairs consisting of synthetic images of models, clothing, and real try-on images, used to train models in the absence of real triplet data (i.e., real photos of the same person wearing different clothes in the same pose).

[0021] 6) Diffusion Model: A deep learning model based on probabilistic generation that generates high-quality images through progressive denoising.

[0022] 7) Definition of task abbreviations: Shop to Model (G2M): Transfer flat lay / hanging photos of clothing to simple model studio photos.

[0023] Shop2Street (G2S): Transferring flat-lay / hanging photos of clothing to street-style portraits with complex backgrounds.

[0024] Model-to-Model (M2M): Transferring clothing from one model to another.

[0025] Street to Street (S2S): Transfers the clothing worn by one person in a street photo to another person in a street photo.

[0026] It should be further noted that a related technology similar to the embodiments of this invention is BooW-VTON. BooW-VTON also uses a pseudo-triple-based training method and learns the G2S task by adding background to the pseudo-triples. However, BooW-VTON still follows a two-stage processing flow. That is, during training, it still heavily relies on the semantic segmentation mask generated by the human body parsing network to indicate "where is the clothing, where is the body," causing error accumulation. Furthermore, BooW-VTON does not utilize the characteristics of pseudo-triples to achieve bootstrapping optimization and can only perform G2M and G2S tasks, unable to handle M2M and S2S tasks.

[0027] Based on this, this invention proposes a model that does not rely on human body parsing masks and clothing deformation modules, directly generating results from the input image, thereby mitigating the error accumulation caused by preprocessing. It also proposes a bootstrapping end-to-end (B-E2E) training method, iteratively generating higher-quality pseudo triples to repeatedly refine the model, enabling it to generate images with more realistic details and fewer artifacts. Furthermore, it constructs a curriculum learning (CrL) pipeline, utilizing the difficulty hierarchy among four tasks and initializing paths with specific model weights to achieve knowledge transfer from simple tasks (G2M) to complex tasks (S2S), thus solving all VTO scenarios with a single solution.

[0028] Reference Figure 1 This invention provides a virtual try-on method based on a bootstrap end-to-end diffusion model, the method comprising the following steps: S100. Based on human images and random clothing, generate synthetic images for training and construct an initial pseudo triplet dataset; Specifically, a person image, a first clothing image, and a second clothing image are selected. The person image and the second clothing image are input into the VTO generator for synthesis to obtain a synthesized image. The synthesized image, the first clothing image, and the person image are combined to construct training triplets. The synthesis steps are repeated to construct an initial pseudo-triplet dataset.

[0029] In this embodiment, images of people are selected from the dataset. and a random outfit Using an initial VTO generator Generate synthetic images In this picture, the figures Wearing clothes Initial generator The choice is flexible. To lower the implementation threshold, this embodiment... This can be an existing deep learning-based VTO model, a simple synthesis method based on traditional image processing techniques (such as thin-plate spline interpolation (TPS) combined with Poisson fusion), or a deep learning-based inpainting technique. Even if the initial synthesized image quality is low (with artifacts or misalignment), subsequent bootstrapping iterative strategies can gradually correct these errors. As a training triplet. Among them, As input, real images This serves as the learning objective for the model. This construct allows the model to learn how to reconstruct a "real image wearing the original clothes" by adding the "original clothes" to a "synthetic image of a character wearing random clothes," thus learning the ability to change clothes. A series of training triples are combined to form an initial pseudo-triple dataset. .

[0030] S200. Input the initial pseudo triplet dataset into the end-to-end diffusion model for training to obtain the trained end-to-end diffusion model. Specifically, the initial pseudo-triple dataset is input into an end-to-end diffusion model, which includes an encoder, a denoising network, and a decoder. Based on the encoder of the end-to-end diffusion model, the initial pseudo-triple dataset is image-encoded into latent variables, yielding latent variables for the person, clothing, and the desired fitting target. A time step is randomly selected and noise is sampled, and this noise is added to the desired fitting target latent variable based on the time step, resulting in a noisy desired fitting target latent variable. Based on the denoising network of the end-to-end diffusion model, the noisy desired fitting target latent variable, the time step, the person latent variable, and the clothing latent variable are input for noise prediction, yielding predicted noise. The L2 loss between the predicted noise and the actual noise is calculated, and the parameters of the denoising network are optimized using gradient descent and backpropagation algorithms to obtain the trained end-to-end diffusion model.

[0031] In this embodiment, for each triple, the model first encodes the image as a latent variable: , , ,in, For latent variables of the character; For clothing as a latent variable; For the latent variable of the trial fitting target. and This can also be referred to as the conditional feature of the diffusion model. There are various ways to inject conditional features into the denoising network; in this embodiment, the following method is used: , , The splicing method.

[0032] Randomly select a time step A noise sample is taken according to the general noise sampling formula of the diffusion model. Next, based on the time step The general noise addition formula of the diffusion model will add noise Add to The above yields the latent variables of the noisy trial fitting target. . The range of values ​​is The general noise sampling formula for the diffusion model is: ,in It follows a standard normal distribution. The general formula for adding noise in the diffusion model is: ϵ, where A series of pre-set cumulative hold factors, decreasing from 1 to near 0, can be configured in various ways. This embodiment uses a linear noise scheduling setting. Furthermore, denoising networks The input to the noise prediction and denoising network is: Finally, the L2 loss of the predicted noise and the actual noise is calculated, and the denoising network is optimized using basic gradient descent and backpropagation algorithms in deep learning. The parameters.

[0033] S300. Generate a new pseudo triplet dataset based on the trained end-to-end diffusion model, and iteratively train the end-to-end diffusion model until the model performance is saturated to obtain the bootstrap end-to-end diffusion model. Specifically, based on the trained end-to-end diffusion model, the human image and random clothing are synthesized to obtain a new synthesized image; a new pseudo triplet dataset is constructed based on the new synthesized image, and the trained end-to-end diffusion model is fine-tuned to obtain a new end-to-end diffusion model; the new synthesized image synthesis step and the fine-tuning step of the trained end-to-end diffusion model are repeated until the model performance is saturated, and a bootstrap end-to-end diffusion model is obtained.

[0034] The process involves inputting a person image and random clothing into a trained end-to-end diffusion model; the encoder of the trained end-to-end diffusion model encodes the person image and random clothing to obtain latent variables for the person image and random clothing; a denoising network based on the trained end-to-end diffusion model randomly samples noise and concatenates the latent variables for the person image and random clothing to denoise, resulting in a denoised concatenated image; and a decoder based on the trained end-to-end diffusion model decodes the denoised concatenated image to obtain a new synthetic image.

[0035] like Figure 3 As shown, using the model trained in the previous round Regenerate higher quality portrait images The specific generation process is as follows: first, E will... Encoded as Next, sample pure noise. ,make and will and splicing as The input; then, using The noise is gradually denoised to obtain (There are various denoising algorithms, including DDPM, DDIM, etc. This special case uses the DDIM algorithm); Finally, D is used to... Decode the image to obtain the person's image. .

[0036] Build a new dataset ,use right Fine-tuning was performed to obtain... .

[0037] It needs to be explained that the model of This indicates that this is the [number]. The model obtained through bootstrapping. Similarly, This indicates that this is the [number]. The model obtained through bootstrapping. of This indicates that it is used for tasks. The model. For example. This indicates that this is a model used for G2M tasks, and that the model has undergone two rounds of bootstrapping training. Indicated by model Based on input The generated image of the person. Indicated by model Based on input The generated image of the person.

[0038] Repeat this process until performance saturates, and obtain... ,in, This indicates the number of bootstrapping training rounds required for the model to reach performance saturation.

[0039] It should be noted that this embodiment of the invention does not include any human parser or warper. This embodiment does not rely on a specific generative network architecture. Any model capable of conditional image generation (e.g., GAN model, Diffusion model) can be used as the generative network of this invention. One application in this embodiment is described below: a U-Net architecture based on a Latent Diffusion Model (such as Stable Diffusion) as the generative network.

[0040] The network includes an encoder E (for extracting latent features from the image) and a denoising network. (Core generation module), decoder D (restores latent features to the image). The network connection is as follows: encoder E compresses the conditional images (image of a person and image of clothing) into the latent space, obtaining the latent variables of the conditional images; within the latent space, the denoising network... The input conditional image latent variable and a piece of pure noise are used to gradually denoise the pure noise to obtain the latent variable of the fitting target. Finally, the decoder D decodes the latent variable of the fitting target into the real image space to obtain the fitting result.

[0041] S400: Design a weight initialization path and a series of pseudo triple generation methods, and train a bootstrap end-to-end diffusion model to realize virtual try-on in multiple scenarios.

[0042] First, it should be noted that, as Figure 4 As shown, this embodiment of the invention designs different ways to generate pseudo triples and the transmission path of model weights according to the nature and difficulty of the task.

[0043] S410. Obtain a standard studio shooting dataset and perform bootstrap iterative training on the bootstrap end-to-end diffusion model to obtain an end-to-end diffusion model that realizes the store-to-model task. Specifically, bootstrapping iterative training was performed using standard studio photography datasets (VITON-HD, DressCode) to obtain... At this stage, the model learned basic clothing texture preservation and human body structure maintenance.

[0044] S420. Initialize and bootstrap iteratively train the end-to-end diffusion model that realizes the store-to-model task in sequence. Replace the background of the person image and the real image with the street scene background to obtain the end-to-end diffusion model that realizes the store-to-street photography task. Specifically, using The parameters are initialized to obtain .use The initial model is used for bootstrapping and iterative training. During the bootstrapping process, when generating pseudo-triples, the human image is used... and real images The background is replaced with the same complex street scene background, allowing the model to adapt to complex backgrounds during learning.

[0045] It should be noted that the bootstrapping processes of the G2S, M2M, and S2S tasks generate pseudo triplet datasets. The method differs from the bootstrapping iterative training on the G2M task, resulting in different generation of pseudo triples.

[0046] S430. Initialize and bootstrap iteratively train the end-to-end diffusion model that realizes the store-to-model task in sequence to generate pseudo triples and obtain the end-to-end diffusion model that realizes the model-to-model task. Specifically, using The parameters are initialized to obtain .use The initial model is used for bootstrapping iterative training. During the bootstrapping process, when generating pseudo-triples, the conditional clothing is no longer a flat lay image, but a model image. As a training triplet. Among them, As input, real images As the target of model learning.

[0047] S440. Initialize and bootstrap iteratively train the end-to-end diffusion model that realizes the store-to-model task. Replace the background of the person image and the real image with the same street scene background, and replace the background of the conditional clothing image with a different street scene background to obtain the end-to-end diffusion model that realizes the street shooting to street shooting task.

[0048] Specifically, using The parameters are initialized to obtain .use The initial model is used for bootstrapping iterative training. During the bootstrapping process, when generating pseudo-triples, the conditional clothing is a model image. This is a training triplet. In this triplet, the image of the person needs to be... and real images Replace the background with the same complex street scene background; change the conditional clothing image. The background is replaced with another, more complex street scene background. After bootstrapping iterative training, the following is obtained: .

[0049] Furthermore, the simulation experiment is explained in conjunction with the attached diagram: First, the model was trained using the VITON-HD and DressCode datasets. Model inference used the VITON-HD, DressCode, and StreetTryOn datasets.

[0050] Base Models: The experiment used two models: Stable Diffusion V1.5 (SD1.5) and Stable Diffusion XL (SDXL). This embodiment does not depend on any specific base model; this experiment is merely an illustration of a specific implementation. Furthermore, to balance performance and efficiency, the number of bootstrapping iterations was set to I=2. The learning rate was 1e-4.

[0051] Effect verification: G2M Task: As shown in Table 1, the proposed B-E2E model achieves state-of-the-art (SOTA) results on VITON-HD. B-E2E (SDXL) achieves a best score of 0.901 on SSIM and 0.059 on LPIPS, while B-E2E (SD1.5) achieves suboptimal results on both SSIM and LPIPS. These results demonstrate B-E2E's superior ability to synthesize structurally consistent and perceptually coherent images. Table 1 shows that on the G2M task, the FID and KID values ​​of various SOTA methods have reached saturation, with small differences between methods. A similar trend is observed in Table 2, where B-E2E consistently achieves best results on SSIM and LPIPS on the DressCode dataset, while maintaining competitiveness in FID and KID. More importantly, existing two-stage methods often produce synthetic images with significant flaws and structural inconsistencies, such as... Figure 5 As shown. In contrast, our B-E2E model generates higher-quality try-on images by mitigating the problem of error accumulation.

[0052] Table 1: Metrics of the G2M task on the VITON-HD dataset ; Table 2: Metrics of G2M tasks on the DressCode dataset ; G2S task: As shown in Table 3, both B-E2E models outperform state-of-the-art competitors, demonstrating the effectiveness and scalability of the proposed method. Figure 6 This further demonstrates that existing state-of-the-art methods suffer from more severe and pronounced error accumulation problems on complex G2S tasks, while B-E2E alleviates this problem.

[0053] Table 3: Metrics of the G2S task on the StreetTryOn dataset ; M2M and S2S tasks: Table 4 shows that for the M2M and S2S tasks, B-E2E significantly improves the FID metric compared to existing state-of-the-art methods on the VITON-HD and StreetTryOn datasets. Figure 7 and Figure 8 This demonstrates that B-E2E can generate high-quality M2M and S2S fitting results.

[0054] Table 4: Metrics for M2M and S2S Tasks ; In summary, the embodiments of the present invention have the following beneficial effects: 1) End-to-end (E2E) generation generates models that do not rely on human body parsing masks and clothing deformation modules, directly generating results from input images, thereby mitigating the accumulation of errors caused by preprocessing.

[0055] 2) The bootstrap end-to-end (B-E2E) training method repeatedly "polishes" the model by iteratively generating higher-quality pseudo triples, enabling it to generate images with more realistic details and fewer artifacts.

[0056] 3) The Course Learning (CrL) pipeline enables knowledge transfer from simple tasks (G2M) to complex tasks (S2S), thus solving all VTO scenarios with a single solution.

[0057] The embodiments of the present invention differ from the prior art in the following technical features: 1) A model that does not rely on human body parsing mask and clothing deformation module is proposed to generate results directly from input image, thereby alleviating the error accumulation caused by preprocessing.

[0058] 2) A bootstrap end-to-end (B-E2E) training method is proposed, which iteratively generates higher-quality pseudo triples to repeatedly "polish" the model, enabling it to generate images with more realistic details and fewer artifacts.

[0059] 3) Construct a course learning (CrL) pipeline, utilize the difficulty hierarchy among the four tasks, initialize the path through specific model weights, and realize knowledge transfer from simple tasks (G2M) to complex tasks (S2S), thereby solving all VTO scenarios with a single solution.

[0060] The embodiments of the present invention have the following advantages over the prior art: 1) Solving the error accumulation problem and improving naturalness: Unlike methods such as BooW-VTON that rely on parsed masks, this invention completely removes the mask input. This means that even with limb occlusion or complex movements, the model will not result in missing or extra pieces of clothing due to incorrect mask drawing. The images generated by the B-E2E model blend more naturally with the skin in details such as collars and cuffs.

[0061] 2) Self-supervised learning without manual data labeling: Through pseudo triples and bootstrapping strategies, this invention can improve model performance by using a large number of paired or single-person images for self-training without expensive data labeling.

[0062] 3) Effective implementation of high-difficulty tasks: Most existing technologies can only handle single, simple tasks such as G2M. This invention, through course learning, uses the M2M model to initialize the S2S model, enabling the model to "remove" clothing from one person and "wear" it on another person in a complex context, which is something that existing single-task models cannot do.

[0063] Reference Figure 2 A virtual try-on system based on a bootstrap end-to-end diffusion model includes: The first module 201 is used to train the generated synthetic images based on human images and random clothing, and to construct the initial pseudo triplet dataset; The second module 202 is used to input the initial pseudo triplet dataset into the end-to-end diffusion model for training, and obtain the trained end-to-end diffusion model. The third module 203 is used to generate a new pseudo triplet dataset based on the trained end-to-end diffusion model, and to iteratively train the end-to-end diffusion model until the model performance is saturated, thus obtaining a bootstrap end-to-end diffusion model. The fourth module 204 is used to design a weight initialization path and a series of pseudo triple generation methods, and to train a bootstrap end-to-end diffusion model to realize virtual try-on in multiple scenarios.

[0064] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0065] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A virtual try-on method based on a bootstrap end-to-end diffusion model, characterized in that, Includes the following steps: Based on human images and random clothing, we train the generated synthetic images to construct an initial pseudo triplet dataset; The initial pseudo triplet dataset is input into the end-to-end diffusion model for training, resulting in the trained end-to-end diffusion model. A new pseudo triplet dataset is generated based on the trained end-to-end diffusion model. The end-to-end diffusion model is iteratively trained until the model performance saturates, resulting in a bootstrap end-to-end diffusion model. Design a weight initialization path and a series of pseudo triple generation methods, and train a bootstrap end-to-end diffusion model to achieve virtual try-on in multiple scenarios.

2. The virtual try-on method based on a bootstrap end-to-end diffusion model according to claim 1, characterized in that, The step of generating synthetic images based on human images and random clothing to construct an initial pseudo-triple dataset specifically includes: Select a person image, a first clothing image, and a second clothing image. Input the person image and the second clothing image into the VTO generator to composite them and obtain a composite image. Training triplets are constructed by combining the synthesized image, the first clothing image, and the character image. The synthesis steps are repeated to construct an initial pseudo-triplet dataset.

3. The virtual try-on method based on a bootstrap end-to-end diffusion model according to claim 2, characterized in that, The step of inputting the initial pseudo-triple dataset into the end-to-end diffusion model for training to obtain the trained end-to-end diffusion model specifically includes: The initial pseudo triplet dataset is input into the end-to-end diffusion model, which includes an encoder, a denoising network, and a decoder. The encoder based on the end-to-end diffusion model encodes the initial pseudo triplet dataset into latent variables, obtaining latent variables of people, clothing, and fitting targets. Randomly select a time step and sample noise, and add the noise to the latent variable of the trial target based on the time step to obtain the noisy trial target latent variable; A denoising network based on an end-to-end diffusion model is used to predict noise by taking input noisy latent variables of the target, time step, person, and clothing. The L2 loss of predicted noise and real noise is calculated, and the parameters of the denoising network are optimized by gradient descent and backpropagation algorithms to obtain the trained end-to-end diffusion model.

4. The virtual try-on method based on a bootstrap end-to-end diffusion model according to claim 3, characterized in that, The step of generating a new pseudo-triple dataset based on the trained end-to-end diffusion model, iteratively training the end-to-end diffusion model until its performance saturates, and obtaining a bootstrap end-to-end diffusion model specifically includes: Based on the trained end-to-end diffusion model, the human image and random clothing are synthesized to obtain a new synthetic image. A new pseudo triplet dataset is constructed based on the new synthetic images, and the trained end-to-end diffusion model is fine-tuned to obtain a new end-to-end diffusion model. The process of iterating through new image synthesis steps and fine-tuning the trained end-to-end diffusion model continues until the model performance saturates, resulting in a bootstrap end-to-end diffusion model.

5. The virtual try-on method based on a bootstrap end-to-end diffusion model according to claim 4, characterized in that, The step of synthesizing a new synthetic image by combining a person image with random clothing based on a trained end-to-end diffusion model specifically includes: Input the image of the person and random clothing into the trained end-to-end diffusion model; The encoder based on the trained end-to-end diffusion model encodes the human image and random clothing to obtain the latent variables of the human image and the latent variables of the random clothing. A denoising network based on a trained end-to-end diffusion model randomly samples noise and concatenates the latent variables of the person image with random clothing latent variables to denoise, resulting in a denoised concatenated image. The decoder based on the trained end-to-end diffusion model decodes the denoised stitched image to obtain a new synthetic image.

6. The virtual try-on method based on a bootstrap end-to-end diffusion model according to claim 5, characterized in that, The step of designing a weight initialization path and a series of pseudo-triple generation methods, and training a bootstrap end-to-end diffusion model to achieve virtual try-on in multiple scenarios, specifically includes: We obtained a standard studio shooting dataset and performed iterative bootstrapping training on the end-to-end diffusion model to obtain an end-to-end diffusion model that realizes the store-to-model task. The end-to-end diffusion model for realizing the store-to-model task is initialized and trained iteratively by bootstrapping. The background of the person image and the real image is replaced with the street scene background to obtain the end-to-end diffusion model for realizing the store-to-street photography task. The end-to-end diffusion model for realizing the store-to-model task is initialized and trained iteratively by bootstrapping to generate pseudo triples, thus obtaining the end-to-end diffusion model for realizing the model-to-model task. The end-to-end diffusion model for realizing the store-to-model task is initialized and trained iteratively by bootstrapping. The backgrounds of the person image and the real image are replaced with the same street scene background, and the backgrounds of the conditional clothing image are replaced with different street scene backgrounds to obtain the end-to-end diffusion model for realizing the street photography to street photography task.

7. A virtual try-on system based on a bootstrap end-to-end diffusion model, characterized in that, Includes the following modules: The first module is used to train the generated synthetic images based on human images and random clothing, and to build an initial pseudo triplet dataset. The second module is used to input the initial pseudo triplet dataset into the end-to-end diffusion model for training, and obtain the trained end-to-end diffusion model. The third module is used to generate a new pseudo triplet dataset based on the trained end-to-end diffusion model, and to iteratively train the end-to-end diffusion model until the model performance is saturated, thus obtaining a bootstrap end-to-end diffusion model. The fourth module is used to design a weight initialization path and a series of pseudo triple generation methods, and to train a bootstrap end-to-end diffusion model to realize virtual try-on in multiple scenarios.

Images