A lightweight virtual try-on method and system based on a diffusion model and a medium

Abstract

The application provides a lightweight virtual fitting method and system based on a diffusion model, and a medium, the method comprising: creating a lightweight mask extractor to extract accurate masks for clothes on a model image and a clothes image; respectively performing shape adjustment on the clothes mask and the clothes image; respectively encoding the model image and the clothes image into a latent space, and using a DDIM inversion technology to respectively obtain T+1 spare noise latent features; in an early step of a diffusion denoising process of the diffusion model, taking a time step t1, and injecting latent information of the spare clothes image into a reconstruction process of the model image; in a later step, taking a time step t1, and restoring the background through a latent information replacement technology; performing step-by-step denoising until complete denoising, and finally decoding the noise-free latent features into a pixel space to obtain a generated refitting image result. The application can generate high-quality fitting images without training the diffusion model, thereby improving user experience.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a lightweight virtual try-on method based on a diffusion model. Background Technology

[0002] The main task of virtual try-on technology is to simulate, through computer simulation, how clothing selected by the user is worn on a photograph provided by the user in a natural manner. Traditional methods typically rely on complex deep learning models, such as Generative Adversarial Networks (GANs), which require long-term training on large-scale datasets to achieve accurate fusion of clothing and human images. While effective, these methods are extremely resource-intensive and time-consuming, making them difficult to apply in resource-constrained environments.

[0003] In recent years, diffusion models, as an emerging generative model, have attracted attention due to their excellent performance in image generation tasks. However, diffusion models also face challenges such as complex training, large data requirements, and high computational costs, which limit their practical applications.

[0004] In summary, current common virtual try-on methods still employ complex GAN or diffusion models. While these methods can generate high-quality images, they often require significant computational resources and a cumbersome training process. Furthermore, these methods typically require users to provide various types of input data, such as human pose maps, segmentation maps, and clothing masks, which is very user-unfriendly for non-professional users and increases the barrier to entry. Therefore, there is an urgent need to find a method that can efficiently implement virtual try-on. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a lightweight virtual try-on method, system and medium based on a diffusion model, so as to solve the problems of excessive resource consumption and complex input in previous virtual try-on methods based on diffusion models.

[0006] In a first aspect, the present invention provides a lightweight virtual try-on method based on a diffusion model, the method comprising:

[0007] Step S1: Create a lightweight mask extractor to extract the precise mask from the clothing on the model image and clothing image, and obtain the model mask and clothing mask;

[0008] Step S2: Adjust the shape of the clothing mask and clothing image respectively, and update the clothing image and clothing mask;

[0009] Step S3: Use a variational autoencoder to process the model image I. m and clothing images I gThey are encoded into the latent space respectively, and T+1 spare noisy latent features are obtained using the DDIM inversion technique.

[0010] Step S4: In the early stage of the diffusion denoising process of the diffusion model, take a time step t1 and inject the potential information of the spare clothing image corresponding to time step t1 into the reconstruction process of the model image.

[0011] Step S5: In the later steps of the diffusion denoising process of the diffusion model, take a time step t2 and restore the background at time step t2 using the latent information replacement technique.

[0012] Step S6: Perform progressive denoising until denoising is complete. Finally, use a variational autoencoder to decode the noise-free latent features into the pixel space to obtain the generated image of the specified model wearing the specified clothing.

[0013] Furthermore, the lightweight mask extractor specifically includes a structure comprising three convolutional layers with ReLU activation functions, a CLIP text encoder, two fully connected layers with activation functions, a 1×1 convolutional layer, a Sigmoid function, and an upsampling module. The mask extraction process is as follows: shallow image features are extracted from the model image or clothing image using three convolutional layers with ReLU activation functions; the text prompt word P is converted into a text embedding using a CLIP text encoder and two fully connected layers with a ReLU activation function in between; then the extracted shallow image features and text embeddings are fused using matrix multiplication and a 1×1 convolutional layer; after activation using the Sigmoid function, the features are interpolated and upsampled to the required resolution to obtain the corresponding mask information.

[0014] Furthermore, step S1 also includes training the lightweight mask extractor to obtain the final lightweight mask extractor, and using clustering to further divide the mask of the model image into upper body mask and lower body mask during the inference process.

[0015] Furthermore, the shape adjustment operation of the clothing mask in step S2 is as follows: calculate the model mask that can be located respectively. The smallest rectangle and clothing mask The smallest rectangle Use image interpolation Interpolate the content in the middle to The size and position of the clothing mask and the model mask are aligned to obtain the clothing mask after shape adjustment. The clothing image is then subjected to the same shape adjustment operation to obtain the adjusted clothing image.

[0016] Furthermore, step S4 specifically includes:

[0017] The noise latent features of the backup model image at time step t = T are used as the starting point for denoising the stable diffusion model. Unet is used for progressive denoising, and when time step t = t1, the clothing latent features corresponding to the backup time step t1 are injected into the latent features of the current inference. The injection position is determined by... The specific calculation formula is as follows:

[0018]

[0019] in, This represents the latent noise variable after injection. This represents the noise latent variable of the model image at time step t1. This represents the potential clothing feature corresponding to the spare time step t1. This indicates a clothing mask.

[0020] Furthermore, step S5 specifically includes: when time step t = t2, injecting the model latent features corresponding to the spare time step t2 into the latent features of the current inference, wherein the injection position is determined by... The specific calculation formula is as follows:

[0021]

[0022] in, This represents the current latent noise variable. This represents the latent features of the model image at time step t2. This represents the latent features of the model corresponding to the spare time step t1. This represents the background mask, which is the complement of the union of the model mask and the clothing mask.

[0023] Secondly, the present invention provides a lightweight virtual try-on system based on a diffusion model, the system comprising:

[0024] The mask extraction module is used to create a lightweight mask extractor to extract precise masks from clothing on model and clothing images, resulting in model and clothing masks.

[0025] The shape adjustment module is used to adjust the shape of the clothing mask and clothing image respectively, and update the clothing image and clothing mask;

[0026] The noise-adding module is used to add noise to the model image I using a variational autoencoder. m and clothing images I g They are encoded into the latent space respectively, and T+1 spare noisy latent features are obtained using the DDIM inversion technique.

[0027] The detail-accurate module is used to take a time step t1 in the early steps of the diffusion denoising process of the diffusion model and inject the potential information of the spare clothing image corresponding to time step t1 into the reconstruction process of the model image.

[0028] The background restoration module is used to take a time step t2 in the later steps of the diffusion denoising process of the diffusion model, and restore the background at time step t2 by latent information replacement technique.

[0029] The costume generation module is used to perform progressive denoising until denoising is complete. Finally, a variational autoencoder is used to decode the noise-free latent features into the image pixel space to obtain the generated costume image of the specified model wearing the specified clothing.

[0030] Furthermore, the lightweight mask extractor specifically includes a structure comprising three convolutional layers with ReLU activation functions, a CLIP text encoder, two fully connected layers with activation functions, a 1×1 convolutional layer, a Sigmoid function, and an upsampling module. The mask extraction process is as follows: shallow image features are extracted from the model image or clothing image using three convolutional layers with ReLU activation functions; the text prompt word P is converted into a text embedding using a CLIP text encoder and two fully connected layers with a ReLU activation function in between; then the extracted shallow image features and text embeddings are fused using matrix multiplication and a 1×1 convolutional layer; after activation using the Sigmoid function, the features are interpolated and upsampled to the required resolution to obtain the corresponding mask information.

[0031] Furthermore, the mask extraction module also includes training a lightweight mask extractor to obtain the final lightweight mask extractor, and using clustering to further divide the mask of the model image into upper body mask and lower body mask during the inference process.

[0032] Furthermore, the shape adjustment operation of the clothing mask is as follows: calculate the model mask that can be located respectively. The smallest rectangle and clothing mask The smallest rectangle Use image interpolation Interpolate the content in the middle to The size and position of the clothing mask and the model mask are aligned to obtain the clothing mask after shape adjustment. The clothing image is then subjected to the same shape adjustment operation to obtain the adjusted clothing image.

[0033] Furthermore, the detailed precision module specifically involves: using the noise latent features of the spare model image at time step t = T as the starting point for denoising the stable diffusion model, performing stepwise denoising using Unet, and when time step t = t1, injecting the clothing latent features corresponding to the spare time step t1 into the latent features of the current inference, with the injection position determined by... The specific calculation formula is as follows:

[0034]

[0035] in, This represents the latent noise variable after injection. This represents the noise latent variable of the model image at time step t1. This represents the potential clothing feature corresponding to the spare time step t1. This indicates a clothing mask.

[0036] Furthermore, the background restoration module specifically comprises:

[0037] When time step t = t2, the model latent features corresponding to the spare time step t2 are injected into the latent features of the current inference, where the injection position is determined by... The specific calculation formula is as follows:

[0038]

[0039] in, This represents the current latent noise variable. This represents the latent features of the model image at time step t2. This represents the latent features of the model corresponding to the spare time step t1. This represents the background mask, which is the complement of the union of the model mask and the clothing mask.

[0040] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0041] The present invention provides one or more technical solutions, which have at least the following technical effects or advantages: by creating a lightweight mask extractor to accurately extract the masks of clothing images and model images, and injecting clothing latent features in the early stage of the denoising process of the diffusion model based on the accurate mask, and injecting background latent information in the later stage of the denoising process, the diffusion model can generate high-quality try-on images without training, while avoiding the introduction of complex input requirements. The present invention can achieve efficient computation of virtual try-on tasks with low resource configuration and show better try-on effects. Attached Figure Description

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] Figure 1 This is an execution flowchart of a lightweight virtual try-on method based on a diffusion model in Embodiment 1 of the present invention;

[0044] Figure 2 This is a schematic diagram of a lightweight virtual try-on system based on a diffusion model according to Embodiment 2 of the present invention;

[0045] Figure 3 This is a schematic diagram of the structure of the medium in Embodiment 3 of the present invention.

[0046] Figure 4 This is a schematic diagram illustrating the lightweight virtual try-on principle of the present invention.

[0047] Figure 5 The diagram shows the effect of the method of the present invention generating results.

[0048] Figure 6 This diagram illustrates the comparison of different fitting effects between the method of this invention and other existing methods under the same experimental data. Detailed Implementation

[0049] This application provides a lightweight virtual try-on method, system, and medium based on a diffusion model to address the problems of existing virtual try-on systems requiring large amounts of computing resources for model training and inference, as well as complex input data requirements that increase the barrier to entry for users.

[0050] The technical solution in this application embodiment follows the following general approach: First, a lightweight CNN network is trained, and the input model image is processed through a lightweight convolutional neural network for shallow feature extraction. Then, the generated shallow features and text embeddings are used for feature fusion to generate a clothing mask, achieving accurate extraction of the clothing mask from both the model and clothing images. Second, latent information injection is performed: in the early stages of the diffusion denoising process, latent information from the clothing image is injected into the reconstruction of the model image to ensure accurate representation of clothing details. Then, background restoration is performed: in the later stages of the diffusion denoising process, the background is restored using latent information replacement technology to ensure the overall consistency of the generated image. This invention, by introducing a lightweight CNN and accurate clothing mask generation technology, combined with a diffusion model, formulates a lightweight virtual try-on method. This method enables efficient computation of virtual try-on tasks with low resource configuration, while generating high-quality try-on images without requiring extensive training of the diffusion model, thus reducing computational resources.

[0051] To better understand the above technical solution, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Example 1

[0053] This embodiment provides a lightweight virtual try-on method based on a diffusion model. Please refer to [link / reference]. Figure 1 and Figure 4 As shown, the method includes:

[0054] Step S1: Create a lightweight mask extractor to extract the precise mask from the clothing on the model image and clothing image, and obtain the model mask and clothing mask;

[0055] Step S2: Adjust the shape of the clothing mask and clothing image respectively, and update the clothing image and clothing mask;

[0056] Step S3: Use a variational autoencoder (VAE) to process the model image. Images of clothing The images are encoded into a latent space, and T+1 alternative noisy latent features are obtained using the DDIM inversion technique, where the model image... The potential noise characteristics of the backup are There are a total of T+1 latent variables, with indices ranging from 0 to T, including clothing images. The potential noise characteristics of the backup are Each time step corresponds to a potential noise feature;

[0057] Step S4: In the early stage of diffusion denoising of the diffusion model, take a time step t1 and inject the potential information of the spare clothing image corresponding to time step t1 into the reconstruction process of the model image.

[0058] Step S5: In the later step of the diffusion denoising process of the diffusion model, take a time step t2 and restore the background at time step t2 using the latent information replacement technique; for example, when T is 50 and the diffusion denoising process starts from t=50 and ends at t=0, the earlier steps include 50-25 and the later steps include 24-0. Preferably, t1 is 40 and t2 is 15.

[0059] Step S6: Perform progressive denoising until denoising is complete. Finally, use a variational autoencoder (VAE) to decode the noise-free latent features into the image pixel space to obtain the generated image of the specified model wearing the specified clothing.

[0060] Preferably, the lightweight mask extractor specifically includes a structure comprising three convolutional layers with ReLU activation functions, a CLIP text encoder, two fully connected layers with activation functions, a 1×1 convolutional layer, a sigmoid function, and an upsampling module. The mask extraction process specifically involves: extracting shallow image features from the model or clothing image using three convolutional layers with ReLU activation functions; using a CLIP text encoder and two fully connected layers with ReLU activation functions in between to convert the text prompt word P into a text embedding; then fusing the extracted shallow image features and text embeddings through matrix multiplication and a 1×1 convolutional layer; activating the feature using a sigmoid function; and finally interpolating and upsampling the result to the required resolution to obtain the corresponding mask information. This lightweight mask extractor enables accurate clothing positioning.

[0061] Preferably, step S1 further includes training the lightweight mask extractor to obtain the final lightweight mask extractor, specifically by using the actual mask value. With mask prediction value The model is trained by calculating the loss function l1, and during inference, clustering is used to further divide the mask of the model image into upper body masks M. up and lower body cover M low This allows for the selection of the appropriate mask as the mask for the model image in different scenarios. Here, the model image corresponds to the image of the clothing worn by the model.

[0062] Preferably, the shape adjustment operation of the clothing mask is as follows: calculate the model mask that can be located respectively. The smallest rectangle and clothing mask The smallest rectangle Use image interpolation Interpolate the content in the middle to The size and position of the clothing mask and the model mask are aligned to obtain the shaped clothing mask. Similarly, the same shape adjustment operation is performed on the clothing image to obtain the adjusted clothing image. That is, by calculating the smallest rectangle of the model image and the smallest rectangle of the clothing image that can be located, image interpolation is used to interpolate the content in the smallest rectangle of the clothing image to the size of the smallest rectangle of the model image, and the size and position of the clothing and the model are aligned to obtain the shaped clothing image.

[0063] Preferably, step S4 specifically includes:

[0064] The noise latent features of the backup model image at time step t = T are used as the starting point for denoising the stable diffusion model. Unet is used for progressive denoising, and when time step t = t1, the clothing latent features corresponding to the backup time step t1 are injected into the latent features of the current inference. The injection position is determined by... The specific calculation formula is as follows:

[0065]

[0066] in, This represents the latent noise variable after injection. This represents the noise latent variable of the model image at time step t1. This represents the potential clothing feature corresponding to the spare time step t1. This indicates a clothing mask.

[0067] Preferably, step S5 specifically includes: when time step t = t2, injecting the model latent features corresponding to the spare time step t2 into the latent features of the current inference, wherein the injection position is determined by... The specific calculation formula is as follows:

[0068]

[0069] in, This represents the current latent noise variable. This represents the latent features of the model image at time step t2. This represents the latent features of the model corresponding to the spare time step t1. This represents the background mask, which is the complement of the union of the model mask and the clothing mask.

[0070] like Figure 5 and Figure 6 As shown, the virtual try-on method based on the SDXL model (i.e., Stable Diffusion XL) of this invention is used. Figure 6 By comparing the diffusion trend in this paper with other existing methods, it can be seen that this method can achieve better qualitative performance in some scenarios compared with other models that use large-scale trial datasets to train on diffusion models. Moreover, this invention does not require training of the diffusion model.

[0071] Based on the same inventive concept, this application also provides a system corresponding to the method in Embodiment 1, as detailed in Embodiment 2.

[0072] Example 2

[0073] This embodiment provides a lightweight virtual try-on system based on a diffusion model, such as Figure 2 and Figure 4 As shown, the system includes:

[0074] The mask extraction module is used to create a lightweight mask extractor to extract precise masks from clothing on model and clothing images, resulting in model and clothing masks.

[0075] The shape adjustment module is used to adjust the shape of the clothing mask and clothing image respectively, and update the clothing image and clothing mask;

[0076] The noise-adding module is used in step S3, where a variational autoencoder (VAE) is used to add noise to the model image. Images of clothing The images are encoded into a latent space, and T+1 alternative noisy latent features are obtained using the DDIM inversion technique, where the model image... The potential noise characteristics of the backup are There are a total of T+1 latent variables, with indices ranging from 0 to T, including clothing images. The potential noise characteristics of the backup are Each time step corresponds to a potential noise feature;

[0077] The detail-accurate module is used to take a time step t1 in the early steps of the diffusion denoising process of the diffusion model and inject the potential information of the spare clothing image corresponding to time step t1 into the reconstruction process of the model image.

[0078] The background restoration module is used to take a time step t2 in the later steps of the diffusion denoising process of the diffusion model, and restore the background at time step t2 by latent information replacement technique.

[0079] The costume generation module is used to perform progressive denoising until denoising is complete. Finally, the variational autoencoder (VAE) is used to decode the noise-free latent features into the pixel space to obtain the generated costume image of the specified model wearing the specified clothing.

[0080] Preferably, the lightweight mask extractor (built using a lightweight CNN network) specifically includes a structure comprising three convolutional layers with ReLU activation functions, a CLIP text encoder, two fully connected layers with activation functions, a 1×1 convolutional layer, a sigmoid function, and an upsampling module. The mask extraction process specifically involves: extracting shallow image features from the model or clothing image using three convolutional layers with ReLU activation functions; using a CLIP text encoder and two fully connected layers with ReLU activation functions in between to convert the text prompt word P into a text embedding; then fusing the extracted shallow image features and text embeddings through matrix multiplication and a 1×1 convolutional layer; activating the feature using a sigmoid function; and finally interpolating and upsampling the result to the required resolution to obtain the corresponding mask information. This lightweight mask extractor enables accurate clothing positioning.

[0081] Preferably, the mask extraction module further includes training a lightweight mask extractor to obtain the final lightweight mask extractor, specifically through the actual mask value. With mask prediction value The model is trained by calculating the loss function l1, and during inference, clustering is used to further divide the mask of the model image into upper body masks M. up and lower body cover M low This allows for the selection of the appropriate mask as the mask for the model image in different scenarios. Here, the model image corresponds to the image of the clothing worn by the model.

[0082] Preferably, the shape adjustment operation of the clothing mask is as follows: calculate the model mask that can be located respectively. The smallest rectangle and clothing mask The smallest rectangle Use image interpolation Interpolate the content in the middle to The size and position of the clothing mask and the model mask are aligned to obtain the shaped clothing mask. Similarly, the same shape adjustment operation is performed on the clothing image to obtain the adjusted clothing image. That is, by calculating the smallest rectangle of the model image and the smallest rectangle of the clothing image that can be located, image interpolation is used to interpolate the content in the smallest rectangle of the clothing image to the size of the smallest rectangle of the model image, and the size and position of the clothing and the model are aligned to obtain the shaped clothing image.

[0083] Preferably, the detailed precision module specifically involves: using the noise latent features of the spare model image at time step t = T as the denoising starting point for the stable diffusion model, performing progressive denoising using Unet, and when time step t = t1, injecting the clothing latent features corresponding to the spare time step t1 into the latent features of the current inference, with the injection position determined by... The specific calculation formula is as follows:

[0084]

[0085] in, This represents the latent noise variable after injection. This represents the noise latent variable of the model image at time step t1. This represents the potential clothing feature corresponding to the spare time step t1. This indicates a clothing mask.

[0086] Preferably, the background restoration module specifically comprises:

[0087] When time step t = t2, the model latent features corresponding to the spare time step t2 are injected into the latent features of the current inference, where the injection position is determined by... The specific calculation formula is as follows:

[0088]

[0089] in, This represents the current latent noise variable. This represents the latent features of the model image at time step t2. This represents the latent features of the model corresponding to the spare time step t1. This represents the background mask, which is the complement of the union of the model mask and the clothing mask.

[0090] Since the system described in Embodiment 2 of the present invention is an apparatus used to implement the method of Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and variations of the apparatus based on the method described in Embodiment 1 of the present invention, and therefore will not be described again here. All apparatuses used in the method of Embodiment 1 of the present invention fall within the scope of protection of the present invention.

[0091] Based on the same inventive concept, this application provides a storage medium corresponding to Embodiment 1, as detailed in Embodiment 3.

[0092] Example 3

[0093] This embodiment provides a computer-readable storage medium, such as... Figure 3As shown, a computer program is stored thereon, which, when executed by a processor, can implement any of the embodiments in Example 1.

[0094] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] The technical solutions provided in this application embodiment have at least the following technical effects or advantages: the present invention can generate high-quality virtual try-on images without retraining the diffusion model by accurately extracting the pre-mask and injecting the latent features twice. The present invention achieves efficient computation of virtual try-on tasks with low resource configuration while maintaining image quality, significantly reducing computational costs and eliminating the need to introduce complex input requirements, thereby providing a more convenient and efficient virtual try-on experience.

[0096] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A lightweight virtual try-on method based on a diffusion model, characterized in that: The method comprises: Step S1, creating a lightweight mask extractor to extract accurate masks for clothes on the model image and the clothes image, to obtain a model mask and a clothes mask; the specific structure of the lightweight mask extractor comprises three convolutional layers+Relu activation function, a CLIP text encoder, two fully connected layers+activation function, a 1x1 convolutional layer, a Sigmoid function and an up-sampling module; the mask extraction process specifically comprises: shallow image feature extraction of the model image or the clothes image is performed through the three convolutional layers+Relu activation function; the text prompt P is converted into a text embedding by using the CLIP text encoder and the two fully connected layers with the structure of the Relu activation function in between; then the extracted shallow image features and the text embedding are fused by matrix multiplication and the 1x1 convolutional layer; after being activated by the Sigmoid function, interpolation up-sampling is performed to the required resolution, and the corresponding mask information is obtained; Step S2, shape adjustment is performed on the clothes mask and the clothes image respectively, and the clothes image and the clothes mask are updated; Step S3, a variational autoencoder is used to encode the model image and the clothes image into a latent space, and a DDIM inversion technology is used to obtain T+1 noise latent features; Step S4, in the pre-step of the diffusion denoising process of the diffusion model, a time step t1 is taken, and the latent information of the standby clothing image corresponding to the time step t1 is injected into the reconstruction process of the model image; specifically, the noise latent feature of the standby model image at the time step t=T is taken as the denoising starting point of the stable diffusion model, Unet is used for step-by-step denoising, and when the time step t=t1, the t1 time step corresponding clothing latent feature of the standby is injected into the current inference latent feature, and the injection position is determined by determination, and the specific calculation formula is as follows: wherein, denotes the noise latent variable after injection, denotes the noise latent variable of the t1 time step mod image, denotes the latent feature of the garment corresponding to the t1 time step, denotes the garment mask; Step S5, taking a time step t2 in the later step of the diffusion denoising process of the diffusion model, restoring the background by latent information replacement technology at time step t2; specifically comprising injecting the model latent feature corresponding to the standby time step t2 into the current inferred latent feature when the time step t=t2, wherein the injection position is determined by Specific calculation formula is as follows: wherein, represents the current noise latent variable, represents the latent features of the t2 time step mannequin image, represents the latent features of the t1 time step mannequin corresponding to the backup, represents the background mask, which is the complement of the union of the mannequin mask and the garment mask; Step S6, step-by-step denoising is performed until complete denoising, and finally the noise-free latent features are decoded into a pixel space by using the variational autoencoder to obtain a generated dressing image result of the specified model wearing the specified clothes.

2. The lightweight virtual try-on method based on diffusion model according to claim 1, characterized in that: In the step S1, the lightweight mask extractor is trained to obtain a final lightweight mask extractor, and in the inference process, the mask of the model image is further divided into an upper body mask and a lower body mask in a clustering manner.

3. The lightweight virtual try-on method based on diffusion model according to claim 1, characterized in that: The shape adjustment operation of the clothing mask in step S2 is as follows: Calculate the model mask that can be located. The smallest rectangle and clothing mask The smallest rectangle Using image interpolation Interpolate the content in the middle to The size and position of the clothing mask and the model mask are aligned to obtain the clothing mask after shape adjustment. The clothing image is then subjected to the same shape adjustment operation to obtain the adjusted clothing image.

4. A lightweight virtual fitting system based on diffusion model, characterized in that: The system comprises: A mask extraction module is configured to create a lightweight mask extractor to extract accurate masks for clothes on the model image and the clothes image, to obtain a model mask and a clothes mask; the specific structure of the lightweight mask extractor comprises three convolutional layers+Relu activation function, a CLIP text encoder, two fully connected layers+activation function, a 1x1 convolutional layer, a Sigmoid function and an up-sampling module; the mask extraction process specifically comprises: shallow image feature extraction of the model image or the clothes image is performed through the three convolutional layers+Relu activation function; the text prompt P is converted into a text embedding by using the CLIP text encoder and the two fully connected layers with the structure of the Relu activation function in between; then the extracted shallow image features and the text embedding are fused by matrix multiplication and the 1x1 convolutional layer; after being activated by the Sigmoid function, interpolation up-sampling is performed to the required resolution, and the corresponding mask information is obtained; A shape adjustment module is configured to perform shape adjustment on the clothes mask and the clothes image respectively, and update the clothes image and the clothes mask; A noise adding module is configured to use a variational autoencoder to encode the model image and the clothes image into a latent space, and use a DDIM inversion technology to obtain T+1 noise latent features. The detail refinement module is used to inject the latent information of the corresponding standby clothing image at a time step t1 into the reconstruction process of the model image in the early step of the diffusion denoising process of the diffusion model. The detail refinement module is specifically: taking the noise latent feature of the standby model image at the time step t=T as the denoising starting point of the stable diffusion model, using Unet for step-by-step denoising, and when the time step t=t1, injecting the t1 time step corresponding clothing latent feature of the standby model into the current inference latent feature, and the injection position is determined by determination, and the specific calculation formula is as follows: wherein, denotes the noise latent variable after injection, denotes the noise latent variable of the t1 time step mod image, denotes the latent feature of the garment corresponding to the t1 time step, denotes the garment mask; a background recovery module configured to take a time step t2 in a later step of the diffusion denoising process of the diffusion model, and recover the background by a latent information replacement technique at the time step t2; the background recovery module specifically comprises: When the time step t = t2, the model latent feature corresponding to the standby time step t2 is injected into the current inference latent feature, wherein the injection position is determined by The specific calculation formula is as follows: wherein, represents the current noise latent variable, represents the latent features of the t2 time step mannequin image, represents the latent features of the t1 time step mannequin corresponding to the backup, represents the background mask, which is the complement of the union of the mannequin mask and the garment mask; a dressing generation module configured to perform step-by-step denoising until complete denoising, and finally decode the noise-free latent feature into a pixel space using a variational autoencoder to obtain a generated dressing image result of the specified model wearing the specified clothes.

5. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1 to 3.

Images