Light-weight virtual fitting method and system based on diffusion model and medium

Abstract

The invention provides a light-weight virtual fitting method and system based on a diffusion model and a medium, and the method comprises the steps: creating a light-weight mask extractor, and extracting precise masks from clothes on a model image and a clothes image; respectively carrying out shape adjustment on the clothing mask and the clothing image; respectively coding the model image and the clothing image into a potential space, and respectively obtaining T + 1 spare noise potential features by using a DDIM inversion technology; taking a time step t1 in the earlier stage of the diffusion denoising process of the diffusion model, and injecting the potential information of the standby clothing image into the reconstruction process of the model image; in the later step, a time step t1 is taken, and the background is recovered through a potential information replacement technology; and step-by-step denoising is carried out until denoising is complete, and finally, noiseless potential features are decoded to a pixel space to obtain a generated reloading image result. According to the method, the high-quality fitting image can be generated without training a diffusion model, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a lightweight virtual try-on method based on a diffusion model. Background Art

[0002] The main task of virtual try-on technology is to wear the user's selected clothing on the user's provided photos in a natural way through computer simulation. Traditional methods usually rely on complex deep learning models, such as generative adversarial networks (GANs), which require long training on large-scale datasets to achieve accurate fusion of clothing and human body images. Although these methods are effective, they have very high requirements for computing resources and time, making them difficult to apply in resource-constrained environments.

[0003] In recent years, diffusion models, as an emerging generative model, have attracted attention due to their outstanding performance in image generation tasks. However, diffusion models also face the problems of complex training, large data requirements and high computational costs, which make them have certain limitations in practical applications.

[0004] In summary, the common virtual try-on methods currently use complex GAN or diffusion models. Although these methods can generate high-quality images, they often require a lot of computing resources and heavy training processes. In addition, these methods usually require users to provide multiple types of input data, such as human posture maps, segmentation maps, and clothing masks, which is very unfriendly to non-professional users and increases the user's usage threshold. In view of this, it is urgent to find a method that can efficiently realize virtual try-on. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a lightweight virtual try-on method, system and medium based on a diffusion model to solve the problems of excessive resource consumption and complex input in the virtual try-on method of the diffusion model in the past.

[0006] In a first aspect, the present invention provides a lightweight virtual try-on method based on a diffusion model, the method comprising:

[0007] Step S1, creating a lightweight mask extractor to extract accurate masks from the clothes on the model image and the clothing image to obtain the model mask and the clothing mask;

[0008] Step S2, adjusting the shapes of the clothing mask and the clothing image respectively, and updating the clothing image and the clothing mask;

[0009] Step S3: Use the variational autoencoder to transform the model image I m and clothing image I gEncode them into the latent space respectively, and use the DDIM inversion technique to obtain T+1 spare noise latent features respectively;

[0010] Step S4, taking a time step t1 in the early step of the diffusion denoising process of the diffusion model, and injecting the potential information of the spare clothing image corresponding to the time step t1 into the reconstruction process of the model image;

[0011] Step S5, taking a time step t2 in the later step of the diffusion denoising process of the diffusion model, and restoring the background at the time step t2 by using the latent information replacement technology;

[0012] Step S6: perform step-by-step denoising until the denoising is complete, and finally use a variational autoencoder to decode the noise-free latent features into a pixel space to obtain a generated dressing-up image result of a specified model wearing specified clothing.

[0013] Furthermore, the specific structure of the lightweight mask extractor includes 3 convolutional layers + Relu activation function, CLIP text encoder, 2 fully connected layers + activation function, 1×1 convolutional layer, Sigmoid function and upsampling module, and the mask extraction process is specifically as follows: extracting shallow image features of model images or clothing images through 3 convolutional layers + Relu activation function, using CLIP text encoder and 2 fully connected layers, with a Relu activation function in the middle, converting the text prompt word P into a text embedding, and then merging the extracted shallow image features and text embedding through matrix multiplication and 1×1 convolutional layer, activating with Sigmoid function, and then interpolating and upsampling to the required resolution to obtain the corresponding mask information.

[0014] Furthermore, the step S1 also includes training the lightweight mask extractor to obtain a final lightweight mask extractor, and further dividing the mask of the model image into an upper body mask and a lower body mask by clustering during the inference process.

[0015] Furthermore, the shape adjustment operation of the clothing mask in step S2 is specifically as follows: respectively calculating the model masks that can be located The smallest rectangle and costume mask The smallest rectangle Use image interpolation to The content in is interpolated to The size of the clothing mask and the model mask is aligned in size and position to obtain the clothing mask after the shape is adjusted, and the clothing image is subjected to the same shape adjustment operation to obtain the adjusted clothing image.

[0016] Furthermore, the step S4 specifically includes:

[0017] The noise potential features of the spare model image at time step t = T are used as the denoising starting point of the stable diffusion model, and Unet is used for step-by-step denoising. When time step t = t1, the spare clothing potential features corresponding to time step t1 are injected into the current inference potential features. The injection position is determined by The specific calculation formula is as follows:

[0018]

[0019] in, represents the noise latent variable after injection, represents the noise latent variable of the model image at time step t1, represents the clothing potential features corresponding to the spare t1 time step, Indicates a clothing mask.

[0020] Further, the step S5 specifically includes: when the time step t=t2, injecting the model potential feature corresponding to the spare time step t2 into the current inference potential feature, wherein the injection position is represented by The specific calculation formula is as follows:

[0021]

[0022] in, represents the current noise latent variable, represents the potential features of the model image at time step t2, represents the model potential features corresponding to the spare t1 time step, represents a background mask, which is the complement of the union of the model mask and the clothing mask.

[0023] In a second aspect, the present invention provides a lightweight virtual try-on system based on a diffusion model, the system comprising:

[0024] The mask extraction module is used to create a lightweight mask extractor to extract accurate masks of clothes on model images and clothing images to obtain model masks and clothing masks;

[0025] A shape adjustment module, used to adjust the shapes of the clothing mask and the clothing image respectively, and update the clothing image and the clothing mask;

[0026] The noise adding module is used to use the variational self-encoder to transform the model image I m and clothing image I g Encode them into the latent space respectively, and use the DDIM inversion technique to obtain T+1 spare noise latent features respectively;

[0027] A detail precision module is used to take a time step t1 in the early step of the diffusion denoising process of the diffusion model, and inject the potential information of the spare clothing image corresponding to the time step t1 into the reconstruction process of the model image;

[0028] A background restoration module is used to take a time step t2 in the later step of the diffusion denoising process of the diffusion model, and restore the background at the time step t2 by using a latent information replacement technique;

[0029] The dressing generation module is used to perform step-by-step denoising until the denoising is complete. Finally, the variational autoencoder is used to decode the noise-free latent features into the image pixel space to obtain the generated dressing image result of the specified model wearing the specified clothes.

[0030] Furthermore, the specific structure of the lightweight mask extractor includes 3 convolutional layers + Relu activation function, CLIP text encoder, 2 fully connected layers + activation function, 1×1 convolutional layer, Sigmoid function and upsampling module, and the mask extraction process is specifically as follows: extracting shallow image features of model images or clothing images through 3 convolutional layers + Relu activation function, using CLIP text encoder and 2 fully connected layers, with a Relu activation function in the middle, converting the text prompt word P into a text embedding, and then merging the extracted shallow image features and text embedding through matrix multiplication and 1×1 convolutional layer, activating with Sigmoid function, and then interpolating and upsampling to the required resolution to obtain the corresponding mask information.

[0031] Furthermore, the mask extraction module also includes training the lightweight mask extractor to obtain a final lightweight mask extractor, and in the inference process, using a clustering method to further divide the mask of the model image into an upper body mask and a lower body mask.

[0032] Furthermore, the shape adjustment operation of the clothing mask is specifically as follows: respectively calculating the model mask that can be located The smallest rectangle and costume mask The smallest rectangle Use image interpolation to The content in is interpolated to The size of the clothing mask and the model mask is aligned in size and position to obtain the clothing mask after the shape is adjusted, and the clothing image is subjected to the same shape adjustment operation to obtain the adjusted clothing image.

[0033] Furthermore, the detailed precision module is specifically as follows: the noise potential features of the spare model image at time step t=T are used as the denoising starting point of the stable diffusion model, Unet is used for step-by-step denoising, and when time step t=t1, the spare clothing potential features corresponding to the t1 time step are injected into the current inference potential features, and the injection position is determined by The specific calculation formula is as follows:

[0034]

[0035] in, represents the noise latent variable after injection, represents the noise latent variable of the model image at time step t1, represents the clothing potential features corresponding to the spare t1 time step, Indicates a clothing mask.

[0036] Furthermore, the background recovery module is specifically:

[0037] When time step t=t2, the model latent features corresponding to the spare time step t2 are injected into the current inference latent features, where the injection position is determined by The specific calculation formula is as follows:

[0038]

[0039] in, represents the current noise latent variable, represents the potential features of the model image at time step t2, represents the model potential features corresponding to the spare t1 time step, represents a background mask, which is the complement of the union of the model mask and the clothing mask.

[0040] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0041] One or more technical solutions provided by the present invention have at least the following technical effects or advantages: by creating a lightweight mask extractor to accurately extract masks from clothing images and model images, and based on the accurate masks, clothing latent features are injected in the early stage of the denoising process of the diffusion model, and background latent information is injected in the later stage of the denoising process, so that the diffusion model can generate high-quality try-on images without training, while avoiding the introduction of complex input requirements. The present invention can achieve efficient calculation of virtual try-on tasks while showing better try-on effects under lower resource configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0043] Figure 1 This is an execution flow chart of a lightweight virtual try-on method based on a diffusion model in Embodiment 1 of the present invention;

[0044] Figure 2 It is a structural schematic diagram of a lightweight virtual try-on system based on a diffusion model in the second embodiment of the present invention;

[0045] Figure 3 Schematic diagram of the structure of the medium in the third embodiment of the present invention.

[0046] Figure 4 It is a schematic diagram of the lightweight virtual try-on principle of the present invention.

[0047] Figure 5 The method of the present invention generates a result display effect diagram.

[0048] Figure 6 This is a schematic diagram comparing different fitting effects between the method of the present invention and other existing methods under the same experimental data. DETAILED DESCRIPTION

[0049] The embodiments of the present application provide a lightweight virtual try-on method, system and medium based on a diffusion model to solve the problem that the existing virtual try-on requires a large amount of computing resources for model training and reasoning, and has complex requirements for input data, which increases the user's usage threshold.

[0050] The technical solution in the embodiment of the present application has the following general idea: first, a lightweight CNN network is trained, and the input model image is subjected to shallow feature extraction through a lightweight convolutional neural network; then, the generated shallow features and text embedding are used for feature fusion to generate clothing masks, so as to achieve accurate extraction of clothing masks on model images and clothing images; secondly, potential information injection is performed: in the early steps of the diffusion denoising process, the potential information of the clothing image is injected into the reconstruction process of the model image to ensure the accurate presentation of clothing details; then background recovery is performed: in the later steps of the diffusion denoising process, the background is restored by the potential information replacement technology to ensure the overall consistency of the generated image. The present invention introduces a lightweight CNN and an accurate clothing mask generation technology, and combines the diffusion model to formulate a lightweight virtual try-on method, which can realize efficient calculation of virtual try-on tasks under lower resource configuration, and generate high-quality try-on images at the same time, without the need to train a large number of diffusion models, thereby reducing computing resources.

[0051] In order to better understand the above technical solution, the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0052] Embodiment 1

[0053] This embodiment provides a lightweight virtual try-on method based on a diffusion model. Figure 1 and Figure 4 As shown, the method includes:

[0054] Step S1, creating a lightweight mask extractor to extract accurate masks from the clothes on the model image and the clothing image to obtain the model mask and the clothing mask;

[0055] Step S2, adjusting the shapes of the clothing mask and the clothing image respectively, and updating the clothing image and the clothing mask;

[0056] Step S3: Use Variational Autoencoder (VAE) to transform the model image and clothing images are encoded into the latent space respectively, and the DDIM inversion technique is used to obtain T+1 spare noise latent features, where the model image The alternative noise potential characteristic is There are T+1 latent variables, with subscripts from 0 to T, clothing images The alternative noise potential characteristic is Each time step corresponds to a noise latent feature;

[0057] Step S4, taking a time step t1 in the early step of diffusion denoising of the diffusion model, and injecting the potential information of the spare clothing image corresponding to the time step t1 into the reconstruction process of the model image;

[0058] Step S5, taking a time step t2 in the later step of the diffusion denoising process of the diffusion model, and restoring the background at time step t2 by using the potential information replacement technology; for example, when T is 50, and the diffusion denoising process starts from t=50 and ends at t=0, then its early steps include 50-25, and the later steps include 24-0. Preferably, t1 is 40 and t2 is 15.

[0059] Step S6: perform step-by-step denoising until the denoising is complete, and finally use a variational autoencoder (VAE) to decode the noise-free latent features into the image pixel space to obtain a generated dressing image result of the specified model wearing the specified clothing.

[0060] Preferably, the specific structure of the lightweight mask extractor includes 3 convolution layers + Relu activation function, CLIP text encoder, 2 fully connected layers + activation function, 1×1 convolution layer, Sigmoid function and upsampling module, and the mask extraction process is specifically as follows: extract shallow image features of model images or clothing images through 3 convolution layers + Relu activation function, use CLIP text encoder and 2 fully connected layers, with Relu activation function in the middle structure, convert text prompt word P into text embedding, then extract shallow image features and text embedding through matrix multiplication and 1×1 convolution layer for feature fusion, after activation with Sigmoid function, interpolate and upsample to the required resolution, that is, obtain corresponding mask information. Accurate clothing positioning can be achieved through this lightweight mask extractor.

[0061] Preferably, the step S1 further includes training the lightweight mask extractor to obtain a final lightweight mask extractor, which can be obtained by training the true value of the mask With mask prediction value The loss function l1 is calculated to train the model, and the mask of the model image is further divided into upper body mask M by clustering during the inference process. up And lower body mask M low , so as to select the corresponding mask as the mask of the model image in different scenes. The model image here corresponds to the clothing image on the model.

[0062] Preferably, the shape adjustment operation of the clothing mask is as follows: respectively calculating the model mask that can be located The smallest rectangle and costume mask The smallest rectangle Use image interpolation to The content in is interpolated to The size of the clothing mask and the model mask is aligned in size and position, thereby obtaining the clothing mask after the shape is adjusted. Similarly, the clothing image performs the same shape adjustment operation to obtain the adjusted clothing image, that is, by respectively calculating the minimum rectangle of the model image and the minimum rectangle of the clothing image that can be located, using image interpolation to interpolate the content in the minimum rectangle of the clothing image to the size of the minimum rectangle of the model image, aligning the size and position of the clothing and the model, thereby obtaining the clothing image after the shape is adjusted.

[0063] Preferably, the step S4 specifically includes:

[0064] The noise potential features of the spare model image at time step t = T are used as the denoising starting point of the stable diffusion model, and Unet is used for step-by-step denoising. When time step t = t1, the spare clothing potential features corresponding to time step t1 are injected into the current inference potential features. The injection position is determined by The specific calculation formula is as follows:

[0065]

[0066] in, represents the noise latent variable after injection, represents the noise latent variable of the model image at time step t1, represents the clothing potential features corresponding to the spare t1 time step, Indicates a clothing mask.

[0067] Preferably, the step S5 specifically includes: when the time step t=t2, injecting the model latent features corresponding to the spare time step t2 into the current inference latent features, wherein the injection position is determined by The specific calculation formula is as follows:

[0068]

[0069] in, represents the current noise latent variable, represents the potential features of the model image at time step t2, represents the model potential features corresponding to the spare t1 time step, represents the background mask, which is the complement of the union of the model mask and the clothing mask, that is,

[0070] like Figure 5 and Figure 6 As shown, the virtual try-on method based on the SDXL model (i.e., Stable Diffusion XL) of the present invention is used. Figure 6 By comparing the Diffusion trend in the above method with other existing methods, it can be seen that the present method can have better qualitative performance in some scenarios compared with other models that use large fitting datasets to perform large-scale training on diffusion models, and the present invention does not require the training of diffusion models.

[0071] Based on the same inventive concept, the present application also provides a system corresponding to the method in Example 1, see Example 2 for details.

[0072] Embodiment 2

[0073] In this embodiment, a lightweight virtual try-on system based on a diffusion model is provided. Figure 2 and Figure 4 As shown, the system comprises:

[0074] The mask extraction module is used to create a lightweight mask extractor to extract accurate masks of clothes on model images and clothing images to obtain model masks and clothing masks;

[0075] A shape adjustment module, used to adjust the shapes of the clothing mask and the clothing image respectively, and update the clothing image and the clothing mask;

[0076] The noise adding module is used in step S3, using a variational autoencoder (VAE) to transform the model image and clothing images are encoded into the latent space respectively, and the DDIM inversion technique is used to obtain T+1 spare noise latent features, where the model image The alternative noise potential characteristic is There are T+1 latent variables, with subscripts from 0 to T, clothing images The alternative noise potential characteristic is Each time step corresponds to a noise latent feature;

[0077] A detail precision module is used to take a time step t1 in the early step of the diffusion denoising process of the diffusion model, and inject the potential information of the spare clothing image corresponding to the time step t1 into the reconstruction process of the model image;

[0078] A background restoration module is used to take a time step t2 in the later step of the diffusion denoising process of the diffusion model, and restore the background at the time step t2 by using a latent information replacement technique;

[0079] The dressing generation module is used to perform step-by-step denoising until the denoising is complete. Finally, the variational autoencoder (VAE) is used to decode the noise-free latent features into the pixel space to obtain the generated dressing image result of the specified model wearing the specified clothes.

[0080] Preferably, the specific structure of the lightweight mask extractor (built using a lightweight CNN network) includes 3 convolutional layers + Relu activation function, CLIP text encoder, 2 fully connected layers + activation function, 1×1 convolutional layer, Sigmoid function and upsampling module, and the mask extraction process is specifically as follows: extract shallow image features of model images or clothing images through 3 convolutional layers + Relu activation function, use CLIP text encoder and 2 fully connected layers, with a structure of Relu activation function in the middle, convert text prompt word P into text embedding, and then perform feature fusion of the extracted shallow image features and text embedding through matrix multiplication and 1×1 convolutional layer, activate with Sigmoid function, and then interpolate and upsample to the required resolution to obtain the corresponding mask information. Accurate clothing positioning can be achieved through this lightweight mask extractor.

[0081] Preferably, the mask extraction module further includes training the lightweight mask extractor to obtain a final lightweight mask extractor, which can be specifically obtained by training the true value of the mask With mask prediction value The loss function l1 is calculated to train the model, and the mask of the model image is further divided into upper body mask M by clustering during the inference process. up And lower body mask M low , so as to select the corresponding mask as the mask of the model image in different scenes. The model image here corresponds to the clothing image on the model.

[0082] Preferably, the shape adjustment operation of the clothing mask is as follows: respectively calculating the model mask that can be located The smallest rectangle and costume mask The smallest rectangle Use image interpolation to The content in is interpolated to The size of the clothing mask and the model mask is aligned in size and position, thereby obtaining the clothing mask after the shape is adjusted. Similarly, the clothing image performs the same shape adjustment operation to obtain the adjusted clothing image, that is, by respectively calculating the minimum rectangle of the model image and the minimum rectangle of the clothing image that can be located, using image interpolation to interpolate the content in the minimum rectangle of the clothing image to the size of the minimum rectangle of the model image, aligning the size and position of the clothing and the model, thereby obtaining the clothing image after the shape is adjusted.

[0083] Preferably, the detailed precision module is specifically as follows: the noise potential features of the spare model image at time step t=T are used as the denoising starting point of the stable diffusion model, Unet is used for stepwise denoising, and when time step t=t1, the spare clothing potential features corresponding to the t1 time step are injected into the current inference potential features, and the injection position is determined by The specific calculation formula is as follows:

[0084]

[0085] in, represents the noise latent variable after injection, represents the noise latent variable of the model image at time step t1, represents the clothing potential features corresponding to the spare t1 time step, Indicates a clothing mask.

[0086] Preferably, the background recovery module is specifically:

[0087] When time step t=t2, the model latent features corresponding to the spare time step t2 are injected into the current inference latent features, where the injection position is determined by The specific calculation formula is as follows:

[0088]

[0089] in, represents the current noise latent variable, represents the potential features of the model image at time step t2, represents the model potential features corresponding to the spare t1 time step, represents the background mask, which is the complement of the union of the model mask and the clothing mask, that is,

[0090] Since the system introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0091] Based on the same inventive concept, the present application provides a storage medium corresponding to Example 1, see Example 3 for details.

[0092] Embodiment 3

[0093] This embodiment provides a computer-readable storage medium, such as Figure 3As shown, a computer program is stored thereon, and when the computer program is executed by a processor, any implementation method in Example 1 can be implemented.

[0094] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages: the present invention can generate high-quality try-on images without retraining the diffusion model through the precise extraction of the pre-mask and the double injection of potential features. The present invention achieves efficient calculation of the virtual try-on task under lower resource configuration while maintaining image quality, significantly reduces the computing cost, and does not need to introduce complex input requirements, thereby providing a more convenient and efficient virtual try-on experience.

[0096] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A lightweight virtual try-on method based on a diffusion model, characterized by: The method comprises: Step S1, creating a lightweight mask extractor to extract accurate masks from the clothes on the model image and the clothing image to obtain the model mask and the clothing mask; Step S2, adjusting the shapes of the clothing mask and the clothing image respectively, and updating the clothing image and the clothing mask; Step S3: using a variational autoencoder, encoding the model image and the clothing image into a latent space respectively, and using the DDIM inversion technique to obtain T+1 spare noise latent features respectively; Step S4, taking a time step t1 in the early step of the diffusion denoising process of the diffusion model, and injecting the potential information of the spare clothing image corresponding to the time step t1 into the reconstruction process of the model image; Step S5, taking a time step t2 in the later step of the diffusion denoising process of the diffusion model, and restoring the background at the time step t2 by using the latent information replacement technology; Step S6: perform step-by-step denoising until the denoising is complete, and finally use a variational autoencoder to decode the noise-free latent features into a pixel space to obtain a generated dressing-up image result of a specified model wearing specified clothing.

2. The lightweight virtual try-on method based on diffusion model according to claim 1, characterized in that: The specific structure of the lightweight mask extractor includes 3 convolutional layers + Relu activation function, CLIP text encoder, 2 fully connected layers + activation function, 1×1 convolutional layer, Sigmoid function and upsampling module. The mask extraction process is specifically as follows: extracting shallow image features of model images or clothing images through 3 convolutional layers + Relu activation function, using CLIP text encoder and 2 fully connected layers with a Relu activation function in the middle, converting the text prompt word P into text embedding, and then fusing the extracted shallow image features and text embedding through matrix multiplication and 1×1 convolutional layer, activating with Sigmoid function, and then interpolating and upsampling to the required resolution to obtain the corresponding mask information.

3. The lightweight virtual try-on method based on diffusion model according to claim 1, characterized in that: The step S1 also includes training the lightweight mask extractor to obtain a final lightweight mask extractor, and further dividing the mask of the model image into an upper body mask and a lower body mask by clustering during the inference process.

4. The lightweight virtual try-on method based on diffusion model according to claim 1, characterized in that: The shape adjustment operation of the clothing mask in step S2 is specifically as follows: respectively calculate the model mask that can be located The smallest rectangle and costume mask The smallest rectangle Use image interpolation to The content in is interpolated to The size of the clothing mask and the model mask is aligned in size and position to obtain the clothing mask after the shape is adjusted, and the clothing image is subjected to the same shape adjustment operation to obtain the adjusted clothing image.

5. The lightweight virtual try-on method based on diffusion model according to claim 1, characterized in that: The step S4 specifically includes: The noise potential features of the spare model image at time step t = T are used as the denoising starting point of the stable diffusion model, and Unet is used for step-by-step denoising. When time step t = t1, the spare clothing potential features corresponding to time step t1 are injected into the current inference potential features. The injection position is determined by The specific calculation formula is as follows: in, represents the noise latent variable after injection, represents the noise latent variable of the model image at time step t1, represents the potential features of clothing corresponding to the spare t1 time step, Indicates a clothing mask.

6. The lightweight virtual try-on method based on diffusion model according to claim 1, characterized in that: The step S5 specifically includes: when the time step t=t2, injecting the model potential feature corresponding to the spare time step t2 into the current inference potential feature, wherein the injection position is represented by The specific calculation formula is as follows: in, represents the current noise latent variable, represents the potential features of the model image at time step t2, represents the model potential features corresponding to the spare t1 time step, represents a background mask, which is the complement of the union of the model mask and the clothing mask.

7. A lightweight virtual try-on system based on a diffusion model, characterized by: The system comprises: The mask extraction module is used to create a lightweight mask extractor to extract accurate masks of clothes on model images and clothing images to obtain model masks and clothing masks; A shape adjustment module, used to adjust the shapes of the clothing mask and the clothing image respectively, and update the clothing image and the clothing mask; A denoising module is used to encode the model image and the clothing image into the latent space respectively using a variational autoencoder, and obtain T+1 spare noise latent features respectively using the DDIM inversion technique; A detail precision module is used to take a time step t1 in the early step of the diffusion denoising process of the diffusion model, and inject the potential information of the spare clothing image corresponding to the time step t1 into the reconstruction process of the model image; A background restoration module, for taking a time step t2 in a later step of the diffusion denoising process of the diffusion model, and restoring the background at the time step t2 by a latent information replacement technique; The dressing generation module is used to perform step-by-step denoising until the denoising is complete. Finally, the variational autoencoder is used to decode the noise-free latent features into the pixel space to obtain the generated dressing image result of the specified model wearing the specified clothes.

8. The lightweight virtual try-on system based on diffusion model according to claim 7, characterized in that: The specific structure of the lightweight mask extractor includes 3 convolutional layers + Relu activation function, CLIP text encoder, 2 fully connected layers + activation function, 1×1 convolutional layer, Sigmoid function and upsampling module. The mask extraction process is specifically as follows: extracting shallow image features of model images or clothing images through 3 convolutional layers + Relu activation function, using CLIP text encoder and 2 fully connected layers with a Relu activation function in the middle, converting the text prompt word P into text embedding, and then fusing the extracted shallow image features and text embedding through matrix multiplication and 1×1 convolutional layer, activating with Sigmoid function, and then interpolating and upsampling to the required resolution to obtain the corresponding mask information.

9. The lightweight virtual try-on system based on diffusion model according to claim 7, characterized in that: The detailed precision module specifically includes: taking the noise potential features of the spare model image at time step t=T as the denoising starting point of the stable diffusion model, using Unet for step-by-step denoising, and when time step t=t1, injecting the spare clothing potential features corresponding to the t1 time step into the current inference potential features, the injection position is determined by The specific calculation formula is as follows: in, represents the noise latent variable after injection, represents the noise latent variable of the model image at time step t1, represents the potential features of clothing corresponding to the spare t1 time step, Indicates a clothing mask; The background recovery module is specifically: When time step t=t2, the model latent features corresponding to the spare time step t2 are injected into the current inference latent features, where the injection position is determined by The specific calculation formula is as follows: in, represents the current noise latent variable, represents the potential features of the model image at time step t2, represents the model potential features corresponding to the spare t1 time step, represents a background mask, which is the complement of the union of the model mask and the clothing mask.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images