Virtual fitting method based on diffusion model, computer equipment and program product

Abstract

The invention relates to a virtual try-on method based on a diffusion model, computer equipment and a program product, and the virtual try-on method comprises the steps: obtaining a clothes graph and a model graph, respectively obtaining a posture graph, a depth graph, a mask graph, model potential features and clothes potential features, and inputting the posture graph, the depth graph, the mask graph, the model potential features and the clothes potential features into a first control network and a second control network; performing convolution on the posture graph and the clothing graph to obtain a posture feature graph, combining the random noise, the model latent features and the posture feature graph, inputting the combination into a first control network, and outputting a posture control vector by the first control network; performing convolution on the depth map and the clothing map to obtain a depth feature map, combining the random noise, the model latent features and the depth feature map, inputting the combination into a second control network, and outputting a depth control vector by the second control network; injecting a posture control vector and a depth control vector into an intermediate layer and a decoder of the first diffusion model to output a fitting image; and comparing the difference loss of the fitting image and the model chart, and training the first control network and the second control network.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and deep learning, and in particular to a virtual try-on method, computer device and program product based on a diffusion model. Background Art

[0002] Image-based virtual try-on is an important computer vision task, whose goal is to generate seamless photos of a person wearing a target garment by inputting a person image and a target garment image. Generating realistic virtual try-on images has attracted widespread interest due to its convenience and potential in providing users with personalized shopping experience in e-commerce. The main challenge of virtual try-on is how to naturally match the garment to the person under different poses and movements, and keep the pattern and texture of the garment undistorted.

[0003] Existing image-based virtual try-on methods mainly rely on generative adversarial networks. These methods first distort the clothing image to match a given person image, and then combine the deformed image with the person image into a generator for synthesis. However, try-on methods based on generative adversarial networks face the following problems: (1) deformation methods cannot handle challenging poses. (2) generated images often lack a certain degree of realism and may not be able to generate finer details.

[0004] Pre-trained diffusion models have shown excellent performance in various vision tasks. Compared with generative adversarial networks, diffusion models excel in generating images with fine-grained realism. When pre-trained diffusion models are applied to virtual try-on tasks, the key issue is how to ensure the controllability of the generated results, especially to maintain the complex textures and patterns of the target clothing. Summary of the invention

[0005] Based on this, it is necessary to provide a virtual try-on method based on a diffusion model to address the above technical issues.

[0006] The virtual fitting method based on the diffusion model of the present application is used to integrate a given clothing image into a target person image to generate a fitting image. The virtual fitting method includes:

[0007] Obtaining a clothing image and a model image wearing the corresponding clothing, respectively obtaining a posture image, a depth image, and a mask image according to the model image, and obtaining a model latent feature by combining the model image and the mask image;

[0008] Extracting clothing latent features of the clothing image and inputting them into the first control network and the second control network at the same time;

[0009] Convolving the posture graph and the clothing graph to obtain a posture feature graph, combining random noise, the model latent features, and the posture feature graph and inputting the combined data into the first control network, the first control network outputting a posture control vector;

[0010] Convolving the depth map with the clothing map to obtain a depth feature map, combining random noise, the model latent features, and the depth feature map, and inputting the combined results into the second control network, where the second control network outputs a depth control vector;

[0011] providing a first diffusion model, inputting the linked clothing image, the model image and the mask image into the first diffusion model, freezing the encoder, the intermediate layer and the decoder of the first diffusion model, and injecting the posture control vector and the depth control vector into the intermediate layer and the decoder, so that the first diffusion model outputs a try-on image;

[0012] The difference loss between the try-on image and the model image is compared to train the first control network and the second control network.

[0013] Optionally, extracting clothing latent features of the clothing image and simultaneously inputting them into the first control network and the second control network specifically includes:

[0014] A feature extractor is used to extract clothing latent features of the clothing image, and the clothing latent features are simultaneously input into the first control network and the second control network respectively through a fully connected layer.

[0015] Optionally, combining the random noise, the model latent features, and the posture feature map and inputting them into the first control network specifically includes:

[0016] The random noise, the model latent features and the mask Figure 3 After being linked, the gesture feature graph is added and calculated, and input into the first control network;

[0017] Combining the random noise, the model latent features, and the depth feature map and inputting them into the second control network specifically includes:

[0018] The random noise, the model latent features and the mask Figure 3 After being linked, it is added with the deep feature map and input into the second control network.

[0019] Optionally, the first control network includes an encoder and an intermediate layer, both of which are to be trained and both are derived from the first diffusion model;

[0020] The second control network includes an encoder and an intermediate layer, both to be trained and both derived from the first diffusion model.

[0021] Optionally, both the intermediate layer and the decoder of the first diffusion model include a frozen self-attention block and a frozen cross-attention block, and a first cross-attention block and a second cross-attention block to be trained are added in the feature path;

[0022] Injecting the posture control vector and the depth control vector into the intermediate layer and the decoder specifically includes:

[0023] Injecting the posture control vector into the first cross-attention block, the posture control vector serving as a first key-value pair for query;

[0024] The depth control vector is injected into the second cross-attention block, and the depth control vector serves as a second key-value pair for query.

[0025] Optionally, the virtual try-on method includes:

[0026] Extracting latent features of the try-on image to obtain a try-on image, upsampling the try-on image to obtain a sampling image of expected resolution, and extracting latent features of the sampling image to obtain a sampling image;

[0027] Inputting random noise, the latent features of the trial-wear image and the latent features of the sampling image into a third control network, and the third control network outputs a control vector;

[0028] inputting random noise into the frozen second diffusion model, injecting the control vector into the second diffusion model, thereby guiding the second diffusion model to output a try-on image of expected resolution;

[0029] Obtain an expected resolution sample image corresponding to the model image, compare the expected resolution sample image and the expected resolution try-on image, and the difference loss between the two, and optimize the third control network.

[0030] Optionally, the third control network includes an encoder and an intermediate layer, both of which are to be trained and both are derived from the second diffusion model.

[0031] Optionally, injecting the control vector into the second diffusion model specifically includes:

[0032] The second diffusion model includes an encoder, an intermediate layer and a decoder, wherein the intermediate layer and the decoder each include a self-attention block and a cross-attention block;

[0033] The control vector is input into the cross-attention block, and the control vector serves as a third key-value pair for query.

[0034] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the virtual try-on method based on the diffusion model described in the present application.

[0035] The present application also provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the virtual try-on method based on the diffusion model described in the present application.

[0036] The virtual try-on method based on the diffusion model of this application has at least the following effects:

[0037] The present application uses a model image recording the model's posture and a clothing image of the target clothing as input, and extracts a posture feature map and a depth feature map from the model image, respectively. The posture feature map and the depth feature map are used as posture control conditions and depth control conditions, respectively, so that the two control networks output posture control variables and depth control variables, respectively. Both the posture control variables and the depth control variables can provide different and effective conditional information for the first stable diffusion model, thereby guiding the first stable diffusion model to generate a natural and realistic fitting image, and while generating it, maintain the complex features in the clothing image, including text, patterns, and style. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of a flow chart of a virtual try-on method based on a diffusion model in an embodiment of the present application;

[0039] Figure 2 This is a multi-conditional diffusion model framework diagram for implementing a virtual try-on method based on a diffusion model in an embodiment of the present application;

[0040] Figure 3 for Figure 2 Model framework diagram of the multi-condition injection module;

[0041] Figure 4 Schematic diagram of a flow chart of a virtual try-on method based on a diffusion model in an embodiment of the present application;

[0042] Figure 5 A super-resolution diffusion model framework diagram for implementing a diffusion model-based virtual try-on method in an embodiment of the present application;

[0043] Figure 6 for Figure 2 The multi-conditional diffusion model, and Figure 5 The overall pipeline model architecture diagram after the super-resolution diffusion model cascade;

[0044] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] The current virtual try-on method uses a control network architecture in a sample-based inpainting diffusion model to control the generation of try-on images in order to preserve the features of the try-on images and improve the realism of the try-on images. This method maintains the characteristics of clothing and improves the naturalness of the try-on images, but for clothing images with complex features, such as complex fonts and patterns, and uncommon styles, it is difficult for the generated try-on images to properly preserve these complex clothing features.

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] See also Figure 1 and Figure 2 An embodiment of the present application provides a virtual try-on method based on a diffusion model, which is used to integrate a given clothing image into a target person image to generate a try-on image. The virtual try-on method includes steps S100 to S600, wherein:

[0048] Step S100, obtaining a clothing image and an image of a model wearing corresponding clothing, obtaining a posture image, a depth image and a mask image according to the model image, and obtaining the model's latent features by combining the model image and the mask image.

[0049] Step S200, extracting clothing latent features of the clothing image and simultaneously inputting them into the first control network and the second control network.

[0050] Step S300, convolving the posture graph and the clothing graph to obtain a posture feature graph, combining random noise, model latent features, and the posture feature graph and inputting them into a first control network, and the first control network outputs a posture control vector.

[0051] Step S400, convolving the depth map and the clothing map to obtain a depth feature map, combining random noise, model latent features, and the depth feature map and inputting them into a second control network, and the second control network outputs a depth control vector.

[0052] Step S500, providing a first diffusion model, inputting the linked clothing image, model image and mask image into the first diffusion model, freezing the encoder, intermediate layer and decoder of the first diffusion model, and injecting posture control vectors and depth control vectors into the intermediate layer and decoder, so that the first diffusion model outputs a try-on image.

[0053] Step S600, comparing the difference loss between the try-on image and the model image, and training the first control network and the second control network.

[0054] In this embodiment, the first diffusion model in step S500, the first control network in step S200 and the second control network cooperate to form a multi-conditional diffusion model for generating a try-on image. The first diffusion model can be, for example, a pre-trained stable diffusion model belonging to a latent diffusion model. The first diffusion model is frozen, and the first control network and the second control network can be trained.

[0055] After training, the multi-conditional diffusion model is used to fuse the input target person image (the input path is equivalent to the model image) and the clothing image (the target clothing image), and integrates the posture control vector and the depth control vector into the denoising Unet network of the first diffusion model using attention. The multi-conditional diffusion model adopts a control network architecture based on the pre-trained stable diffusion model, while introducing additional control conditions and retaining the generation ability of the stable diffusion model. The additional control conditions include posture control vectors and depth control vectors.

[0056] This embodiment uses a model image recording the model's posture and a clothing image of the target clothing as input, and extracts a posture feature map and a depth feature map from the model image, respectively. The posture feature map and the depth feature map are used as posture control conditions and depth control conditions, respectively, so that the two control networks output posture control variables Co and depth control variables Cd, respectively. Both the posture control variable Co and the depth control variable Cd can provide different and effective condition information for the first stable diffusion model. Thereby guiding the first stable diffusion model to generate a natural and realistic fitting image, and maintaining the complex features in the clothing image including text, patterns, and style, etc. during the generation.

[0057] See also Figure 1 and Figure 2 In some embodiments, optional embodiments and sub-steps of steps S100 to S600 are introduced.

[0058] In step S100, the model latent features are obtained by combining the model image and the mask image, specifically including performing a multiplication operation on the model image and the mask image to obtain the model latent features. It can be understood that the mask image is used to retain the portrait information in the model image.

[0059] Step S200, extracting clothing latent features of the clothing image and simultaneously inputting them into the first control network and the second control network, specifically comprising: extracting clothing latent features of the clothing image using a feature extractor, and simultaneously inputting the clothing latent features into the first control network and the second control network respectively via a fully connected layer.

[0060] The feature extractor in step S200 may select a computer vision self-supervised model, such as DINO-V2. The feature extractor is used to provide detailed feature information for the model to better control the generation of the try-on image.

[0061] In step S300, random noise, model latent features, and posture feature maps are combined and input into the first control network, specifically including: random noise, model latent features, and mask Figure 3 After being linked, it is summed with the posture feature map and input into the first control network.

[0062] In step S400, random noise, model latent features, and depth feature maps are combined and input into a second control network, specifically including: random noise, model latent features, and mask Figure 3 After being linked, it is summed with the deep feature map and input into the second control network.

[0063] The two control networks in step S200 are both derived from the first diffusion model in step S400. The first control network includes an encoder and an intermediate layer, both of which are to be trained and are derived from the first diffusion model. The second control network includes an encoder and an intermediate layer, both of which are to be trained and are derived from the first diffusion model. This embodiment locks all parameters of the first diffusion model, and copies the parameters of the first diffusion model encoder block and the first diffusion model intermediate block (intermediate layer) to the control network, thereby obtaining the first control network and the second control network respectively.

[0064] In step S500, the first diffusion model outputs the try-on image specifically through a VQGAN decoder, and the VQGAN decoder is used to map the latent space to the image space to obtain the try-on image.

[0065] See also Figure 2 and Figure 3 In step S500, both the intermediate layer and the decoder of the first diffusion model include a frozen self-attention block and a frozen cross-attention block, and a first cross-attention block and a second cross-attention block to be trained are added in the feature path. The posture control vector and the depth control vector are injected into the intermediate layer and the decoder, specifically including: injecting the posture control vector into the first cross-attention block, the posture control vector is used as the first key-value pair for query; injecting the depth control vector into the second cross-attention block, the depth control vector is used as the second key-value pair for query.

[0066] Specifically, in order to better integrate the posture control vector and the depth control vector into the frozen pre-trained diffusion model, this embodiment designs a multi-control integration module, which includes: a trainable first cross-attention block (posture attention block), a trainable second cross-attention block (depth attention block), a frozen self-attention block (belonging to the original first stable model), and a cross-attention block (belonging to the original first stable model) arranged along the feature path.

[0067] like Figure 3As shown, the multi-control injection module enhances the original attention modules (including the original self-attention module and the original cross-attention module) in the first diffusion model denoising U-net by introducing trainable pose attention and deep attention, while keeping the self-attention and cross-attention components frozen. The benefit of this structure is that it can enhance all attention modules in the intermediate and decoder applied to the denoising U-net. Specifically, for pose attention, the query vector Q is derived from the latent features of the first diffusion model denoising U-Net, and the key-value pairs (key K and value V) are derived from the pose control vector. For deep attention, the query vector Q is derived from the latent features of the first diffusion model denoising U-Net, and the key-value pairs (key K and value V) are derived from the deep control vector. For frozen cross-attention, this project uses CLIP embedding (Contrastive Language-Image Pre-training) as the key K and value V. CLIP embedding is specifically obtained by inputting the clothing image into the CLIP image encoder.

[0068] During training, this project specifically performs gradient updates on the parameters of pose attention and deep attention in the control network and multi-control injection modules.

[0069]

[0070] In the formula,

[0071] ∈, represents the real noise;

[0072] ∈ θ , represents the prediction noise at time step t;

[0073] x t , represents random noise;

[0074] F(x m ), represents the model latent features, F represents feature extraction, x m Represents the model image multiplied by the mask image;

[0075] m, represents the mask image;

[0076] g, represents the clothing picture;

[0077] d, represents the depth control vector, i.e. Cd;

[0078] o, represents the posture control vector, i.e., Co;

[0079] t represents the time step.

[0080] Steps S100 to S600 generate natural fitting images with complex features through a multi-condition control model, but the resolution of the fitting images often cannot meet expectations. The current virtual fitting method can only generate fitting images with a maximum resolution of 1K, while in actual application scenarios, it may be necessary to generate fitting images with a higher resolution, such as 2K. The most direct method is to use a natural image stable diffusion model to upsample the generated fitting images, but directly using existing super-resolution methods may cause distortion of the upsampled fitting images.

[0081] See also Figure 4 and Figure 5 Based on this, in some embodiments, a super-resolution diffusion model for virtual try-on is provided, and a virtual try-on method based on the diffusion model is specifically recorded in steps S700 to S900, which is used to upsample the low-resolution try-on image to a high-resolution image and maintain the original texture features. Among them:

[0082] Step S700: extracting latent features of the try-on image, upsampling the try-on image to obtain a sampling image of expected resolution, and extracting latent features of the sampling image.

[0083] Step S800: random noise, latent features of the trial image, and latent features of the sampling image are input into a third control network, and the third control network outputs a control vector.

[0084] Step S900, random noise is input into the frozen second diffusion model, and the control vector is injected into the second diffusion model, so as to guide the second diffusion model to output a fitting image of expected resolution, obtain a sample image of expected resolution corresponding to the model image, compare the sample image of expected resolution and the fitting image of expected resolution, and the difference loss between the two, and optimize the third control network.

[0085] In step S700 to step S900: the try-on image (or low-resolution try-on image) of step S600 is upsampled to obtain a sampling image of the expected resolution. At this time, the obtained sampling image lacks image details. The latent features of the try-on image are extracted by DINO-V2, and encoded into the latent space of the third control network through the fully connected layer to obtain the control vector output by the third control network. This project uses DINO-V2 as a feature extractor for low-resolution images in the control network. Compared with CLIP extraction, DINO-V2 encodes the image not only as a global tag, but also as a patch token, which helps to retain the information of the low-resolution image and provide a detailed representation.

[0086] In step S800, the third control network includes an encoder and an intermediate layer, both of which are to be trained and are derived from the second diffusion model. The second diffusion model can be a pre-trained stable natural image diffusion model. The second diffusion model combined with the third control network is a super-resolution diffusion model.

[0087] This embodiment uses the latent features of the fitting image and the latent features of the sampling image based on the fitting image, and provides prior knowledge about the model and the clothes based on the third control network, so that the second diffusion model has pixel controllability. The input of the control vector is derived from the fitting image, which can be used to supervise the details of the fitting image. In this way, the content details of the fitting image are maintained during the process of reconstructing the fitting image into a high-resolution image.

[0088] In step S900, the control vector is injected into the second diffusion model, specifically including: the second diffusion model includes an encoder, an intermediate layer and a decoder, and the intermediate layer and the decoder both include a frozen self-attention block and a frozen cross-attention block; the control vector is input into the cross-attention block, and the control vector is used as a third key-value pair (key K and value V) for query, replacing the original CLIP embedding. Step S900 uses cross-attention to inject the control vector into the Unet network of the frozen second diffusion model. See the following formula for details:

[0089]

[0090] In the formula,

[0091] ∈, represents the real noise;

[0092] ∈ θ , represents the prediction noise;

[0093] z t , represents random noise;

[0094] F(LR) represents the latent features of the sampled image extracted (using the VQGAN encoder);

[0095] c, represents the control vector;

[0096] t represents the time step.

[0097] During inference, the generated low-resolution try-on images are input as conditions into the fine-tuned diffusion model. After t iterations, high-resolution try-on images with high fidelity and detailed clothing features are obtained.

[0098] See also Figure 6, each embodiment of the present application provides a cascade diffusion model, which can improve the controllability of the generation result of the virtual try-on image and generate a high-resolution try-on image. The cascade diffusion model includes a multi-conditional diffusion model and a super-resolution diffusion model. The former is used to generate a try-on image with complex feature details, and the latter is used to improve the resolution of the try-on image and maintain the image details.

[0099] Specifically, the multi-conditional diffusion model is used to generate low-resolution fitting images, and the generated fitting images are kept to contain the complex features of clothing by injecting the posture control vector and the depth control vector into the denoising U-net. The super-resolution diffusion model is mainly composed of a frozen first diffusion model and a trainable third control network, which is used to upsample the generated low-resolution results to high resolution. The embodiments of the present application can integrate a given clothing image onto a target person and generate a realistic high-resolution fitting image.

[0100] It should be understood that although Figure 1 and Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 and Figure 4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0101] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a virtual try-on method based on a diffusion model is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0102] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0103] Step S100, obtaining a clothing image and an image of a model wearing corresponding clothing, obtaining a posture image, a depth image and a mask image according to the model image, and obtaining the model's latent features by combining the model image and the mask image.

[0104] Step S200, extracting clothing latent features of the clothing image and simultaneously inputting them into the first control network and the second control network.

[0105] Step S300, convolving the posture graph and the clothing graph to obtain a posture feature graph, combining random noise, model latent features, and the posture feature graph and inputting them into a first control network, and the first control network outputs a posture control vector.

[0106] Step S400, convolving the depth map and the clothing map to obtain a depth feature map, combining random noise, model latent features, and the depth feature map and inputting them into a second control network, and the second control network outputs a depth control vector.

[0107] Step S500, providing a first diffusion model, inputting the linked clothing image, model image and mask image into the first diffusion model, freezing the encoder, intermediate layer and decoder of the first diffusion model, and injecting posture control vectors and depth control vectors into the intermediate layer and decoder, so that the first diffusion model outputs a try-on image.

[0108] Step S600, comparing the difference loss between the try-on image and the model image, and training the first control network and the second control network.

[0109] In one embodiment, a computer program product is provided, comprising computer instructions, which when executed by a processor implement the following steps:

[0110] Step S100, obtaining a clothing image and an image of a model wearing corresponding clothing, obtaining a posture image, a depth image and a mask image according to the model image, and obtaining the model's latent features by combining the model image and the mask image.

[0111] Step S200, extracting clothing latent features of the clothing image and simultaneously inputting them into the first control network and the second control network.

[0112] Step S300, convolving the posture graph and the clothing graph to obtain a posture feature graph, combining random noise, model latent features, and the posture feature graph and inputting them into a first control network, and the first control network outputs a posture control vector.

[0113] Step S400, convolving the depth map and the clothing map to obtain a depth feature map, combining random noise, model latent features, and the depth feature map and inputting them into a second control network, and the second control network outputs a depth control vector.

[0114] Step S500, providing a first diffusion model, inputting the linked clothing image, model image and mask image into the first diffusion model, freezing the encoder, intermediate layer and decoder of the first diffusion model, and injecting posture control vectors and depth control vectors into the intermediate layer and decoder, so that the first diffusion model outputs a try-on image.

[0115] Step S600, comparing the difference loss between the try-on image and the model image, and training the first control network and the second control network.

[0116] In this embodiment, the computer program product includes a program code portion for executing the steps of the virtual try-on method based on the diffusion model in each embodiment of the present application when the computer program product is executed by one or more computing devices. The computer program product may be stored on a computer-readable recording medium. The computer program product may also be provided for download via a data network (e.g., via a RAN, via the Internet and / or via an RBS). Alternatively or additionally, the method may be encoded in a field programmable gate array (FPGA) and / or an application specific integrated circuit (ASIC), or the functionality may be provided for download with the aid of a hardware description language.

[0117] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0118] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. When the technical features in different embodiments are embodied in the same figure, it can be regarded that the figure also discloses the combination examples of the various embodiments involved.

[0119] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A virtual try-on method based on a diffusion model, characterized in that: For integrating a given clothing image into a target person image to generate a fitting image, the virtual fitting method comprises: Obtaining a clothing image and a model image wearing the corresponding clothing, respectively obtaining a posture image, a depth image, and a mask image according to the model image, and obtaining a model latent feature by combining the model image and the mask image; Extracting clothing latent features of the clothing image and inputting them into the first control network and the second control network at the same time; Convolving the posture graph and the clothing graph to obtain a posture feature graph, combining random noise, the model latent features, and the posture feature graph and inputting the combined data into the first control network, the first control network outputting a posture control vector; Convolving the depth map with the clothing map to obtain a depth feature map, combining random noise, the model latent features, and the depth feature map, and inputting the combined results into the second control network, where the second control network outputs a depth control vector; providing a first diffusion model, inputting the linked clothing image, the model image and the mask image into the first diffusion model, freezing the encoder, the intermediate layer and the decoder of the first diffusion model, and injecting the posture control vector and the depth control vector into the intermediate layer and the decoder, so that the first diffusion model outputs a try-on image; The difference loss between the try-on image and the model image is compared to train the first control network and the second control network.

2. The virtual try-on method based on diffusion model as claimed in claim 1, characterized in that: Extracting clothing latent features of the clothing image and inputting them into the first control network and the second control network at the same time, specifically includes: A feature extractor is used to extract clothing latent features of the clothing image, and the clothing latent features are simultaneously input into the first control network and the second control network respectively through a fully connected layer.

3. The virtual try-on method based on diffusion model as claimed in claim 1, characterized in that: Combining the random noise, the model latent features, and the posture feature map and inputting them into the first control network specifically includes: After linking the random noise, the model latent features and the mask map, the three are added together with the posture feature map and input into the first control network; Combining the random noise, the model latent features, and the depth feature map and inputting them into the second control network specifically includes: After random noise, the model latent features and the mask map are linked, they are added together with the deep feature map and input into the second control network.

4. The virtual try-on method based on diffusion model as claimed in claim 1, characterized in that: The first control network includes an encoder and an intermediate layer, both to be trained and both derived from a first diffusion model; The second control network includes an encoder and an intermediate layer, both to be trained and both derived from the first diffusion model.

5. The virtual try-on method based on diffusion model as claimed in claim 1, characterized in that: The intermediate layer and the decoder of the first diffusion model both include a frozen self-attention block and a frozen cross-attention block, and a first cross-attention block and a second cross-attention block to be trained are added in the feature path; Injecting the posture control vector and the depth control vector into the intermediate layer and the decoder specifically includes: Injecting the posture control vector into the first cross-attention block, the posture control vector serving as a first key-value pair for query; The depth control vector is injected into the second cross-attention block, and the depth control vector serves as a second key-value pair for query.

6. The virtual try-on method based on diffusion model as claimed in claim 1, characterized in that: The virtual try-on method comprises: Extracting latent features of the try-on image to obtain a try-on image, upsampling the try-on image to obtain a sampling image of expected resolution, and extracting latent features of the sampling image to obtain a sampling image; Inputting random noise, the latent features of the trial-wear image and the latent features of the sampling image into a third control network, and the third control network outputs a control vector; inputting random noise into the frozen second diffusion model, injecting the control vector into the second diffusion model, thereby guiding the second diffusion model to output a try-on image of expected resolution; Obtain an expected resolution sample image corresponding to the model image, compare the expected resolution sample image and the expected resolution try-on image, and the difference loss between the two, and optimize the third control network.

7. The virtual try-on method based on diffusion model as claimed in claim 6, characterized in that: The third control network includes an encoder and an intermediate layer, both to be trained and both derived from the second diffusion model.

8. The virtual try-on method based on diffusion model as claimed in claim 7, characterized in that: Injecting the control vector into the second diffusion model specifically includes: The second diffusion model includes an encoder, an intermediate layer and a decoder, wherein the intermediate layer and the decoder each include a self-attention block and a cross-attention block; The control vector is input into the cross-attention block, and the control vector serves as a third key-value pair for query.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the virtual try-on method based on the diffusion model according to any one of claims 1 to 8.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the virtual try-on method based on the diffusion model as described in any one of claims 1 to 8 are implemented.

Images