Virtual fitting method based on diffusion model, computer device and program product

Abstract

The application relates to a virtual fitting method based on a diffusion model, a computer device and a program product, the virtual fitting method comprising the following steps: obtaining a clothing image and a model image, respectively obtaining a pose image, a depth image, a mask image, a model latent feature and a clothing latent feature, and simultaneously inputting the images into a first control network and a second control network; convolving the pose image and the clothing image to obtain a pose feature map, combining random noise, the model latent feature and the pose feature map, and inputting the combined result into the first control network, so that the first control network outputs a pose control vector; convolving the depth image and the clothing image to obtain a depth feature map, combining random noise, the model latent feature and the depth feature map, and inputting the combined result into the second control network, so that the second control network outputs a depth control vector; injecting the pose control vector and the depth control vector into an intermediate layer and a decoder of a first diffusion model to output a fitting image; comparing a difference loss of the fitting image and the model image, and training the first control network and the second control network.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and deep learning, and in particular to a virtual try-on method, computer device, and program product based on a diffusion model. Background Art

[0002] Image-based virtual try-on is an important computer vision task. Its goal is to generate seamless photos of a person wearing a target garment, given an input image of the person and an image of the target garment. Generating realistic virtual try-on images has attracted widespread interest due to its convenience and potential for providing users with a personalized shopping experience in e-commerce. The main challenge in virtual try-on is how to naturally match garments to people in different poses and movements, while preserving the garment's pattern and texture.

[0003] Existing image-based virtual try-on methods mainly rely on generative adversarial networks. These methods first distort the clothing image to match a given person image, and then combine the deformed image with the person image into a generator for synthesis. However, try-on methods based on generative adversarial networks face the following problems: (1) The deformation method cannot handle challenging poses. (2) The generated images often lack a certain degree of realism and may not be able to generate finer details.

[0004] Pre-trained diffusion models have demonstrated excellent performance on various vision tasks. Compared to generative adversarial networks, diffusion models excel in generating images with fine-grained realism. When pre-trained diffusion models are applied to virtual try-on tasks, a key challenge is ensuring the controllability of the generated results, particularly preserving the complex textures and patterns of the target garment. Summary of the Invention

[0005] Based on this, it is necessary to provide a virtual try-on method based on the diffusion model to address the above technical issues.

[0006] This application proposes a virtual try-on method based on a diffusion model, which is used to integrate a given clothing image into a target person image to generate a try-on image. The virtual try-on method includes:

[0007] Obtaining a clothing image and an image of a model wearing the corresponding clothing, obtaining a posture image, a depth image, and a mask image based on the model image, and combining the model image and the mask image to obtain model latent features;

[0008] Extracting clothing latent features from the clothing image and inputting them into the first control network and the second control network at the same time;

[0009] Convolving the posture graph and the clothing graph to obtain a posture feature graph, combining random noise, the model latent features, and the posture feature graph and inputting the combined data into the first control network, the first control network outputting a posture control vector;

[0010] Convolving the depth map with the clothing image to obtain a depth feature map, combining random noise, the model latent features, and the depth feature map, and inputting the resultant data into the second control network, which outputs a depth control vector;

[0011] providing a first diffusion model, inputting the linked garment image, the model image, and the mask image into the first diffusion model, freezing the encoder, intermediate layers, and decoder of the first diffusion model, and injecting the posture control vector and the depth control vector into the intermediate layers and decoder, so that the first diffusion model outputs a try-on image;

[0012] The first control network and the second control network are trained by comparing the difference loss between the try-on image and the model image.

[0013] Optionally, extracting clothing latent features from the clothing image and simultaneously inputting them into the first control network and the second control network specifically includes:

[0014] A feature extractor is used to extract clothing latent features of the clothing image, and the clothing latent features are simultaneously input into the first control network and the second control network through a fully connected layer.

[0015] Optionally, combining random noise, the model latent features, and the posture feature map and inputting them into the first control network specifically includes:

[0016] The random noise, the model latent features and the mask Figure Three After being linked, the sum of the two graphs is calculated with the posture feature graph and input into the first control network;

[0017] Combining random noise, the model latent features, and the depth feature map and inputting the combined results into the second control network specifically includes:

[0018] The random noise, the model latent features and the mask Figure Three After being linked, the two are summed with the depth feature map and input into the second control network.

[0019] Optionally, the first control network includes an encoder and an intermediate layer, both of which are to be trained and are derived from the first diffusion model;

[0020] The second control network includes an encoder and an intermediate layer, both to be trained and both derived from the first diffusion model.

[0021] Optionally, both the intermediate layer and the decoder of the first diffusion model include frozen self-attention blocks and frozen cross-attention blocks, and a first cross-attention block and a second cross-attention block to be trained are added in the feature path;

[0022] Injecting the posture control vector and the depth control vector into the intermediate layer and the decoder specifically includes:

[0023] Injecting the posture control vector into the first cross-attention block, the posture control vector serving as a first key-value pair for query;

[0024] The depth control vector is injected into the second cross-attention block, and the depth control vector serves as a second key-value pair for query.

[0025] Optionally, the virtual try-on method includes:

[0026] Extracting latent features of the try-on image to obtain a try-on image, upsampling the try-on image to obtain a sampling image of expected resolution, and extracting latent features of the sampling image to obtain a sampling image;

[0027] Inputting random noise, the latent features of the try-on image, and the latent features of the sampling image into a third control network, the third control network outputting a control vector;

[0028] inputting random noise into the frozen second diffusion model, and injecting the control vector into the second diffusion model, thereby guiding the second diffusion model to output a try-on image of a desired resolution;

[0029] Obtain an expected resolution sample image corresponding to the model image, compare the expected resolution sample image with the expected resolution try-on image, and calculate the difference loss between the two, and optimize the third control network.

[0030] Optionally, the third control network includes an encoder and an intermediate layer, both of which are to be trained and both are derived from the second diffusion model.

[0031] Optionally, injecting the control vector into the second diffusion model specifically includes:

[0032] The second diffusion model includes an encoder, an intermediate layer, and a decoder, wherein the intermediate layer and the decoder each include a self-attention block and a cross-attention block;

[0033] The control vector is input to the cross-attention block, and the control vector serves as a third key-value pair for query.

[0034] The present application also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the virtual try-on method based on the diffusion model described in the present application.

[0035] The present application also provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the virtual try-on method based on the diffusion model described in the present application.

[0036] The virtual try-on method based on the diffusion model in this application has at least the following effects:

[0037] This application uses a model image that records the model's posture and a clothing image of the target clothing as input, and extracts a posture feature map and a depth feature map from the model image. The posture feature map and the depth feature map serve as posture control conditions and depth control conditions, respectively, so that the two control networks output posture control variables and depth control variables, respectively. Both the posture control variables and the depth control variables can provide different and effective conditional information for the first stable diffusion model, thereby guiding the first stable diffusion model to generate a natural and realistic fitting image, while maintaining the complex features in the clothing image, including text, patterns, and style, while generating it. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of a flow chart of a virtual try-on method based on a diffusion model in one embodiment of the present application;

[0039] Figure 2 This is a multi-conditional diffusion model framework diagram for implementing a diffusion model-based virtual try-on method in one embodiment of the present application;

[0040] Figure 3 for Figure 2 Model framework diagram of the multi-condition injection module;

[0041] Figure 4 Schematic diagram of a flow chart of a virtual try-on method based on a diffusion model in one embodiment of the present application;

[0042] Figure 5 This is a framework diagram of a super-resolution diffusion model for implementing a virtual try-on method based on a diffusion model in one embodiment of the present application;

[0043] Figure 6 for Figure 2 The multi-conditional diffusion model, and Figure 5 The overall pipeline model architecture diagram after the super-resolution diffusion model cascade;

[0044] Figure 7 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] The current virtual try-on method uses a control network architecture in a sample-based restoration diffusion model for controlling the generation of a try-on image to preserve the features of the try-on image and improve the realism of the try-on image. This method maintains the characteristics of the clothes and improves the naturalness of the try-on image, but for clothes images with complex features, such as complex fonts and patterns, and unusual styles, the generated try-on image is difficult to properly preserve these complex clothing features.

[0046] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0047] Referring to Figure 1 and Figure 2 An embodiment of the present application provides a virtual try-on method based on a diffusion model for integrating a given clothes image into a target person image to generate a try-on image, the virtual try-on method comprising steps S100-S600, wherein:

[0048] In step S100, a clothes image and a model image wearing corresponding clothes are obtained, a pose image, a depth image and a mask image are obtained according to the model image, and a model latent feature is obtained by combining the model image and the mask image.

[0049] In step S200, a clothes latent feature of the clothes image is extracted and input into a first control network and a second control network.

[0050] In step S300, a pose feature map is obtained by convolving the pose image and the clothes image, random noise, the model latent feature and the pose feature map are combined and input into the first control network, and the first control network outputs a pose control vector.

[0051] In step S400, a depth feature map is obtained by convolving the depth image and the clothes image, random noise, the model latent feature and the depth feature map are combined and input into the second control network, and the second control network outputs a depth control vector.

[0052] In step S500, a first diffusion model is provided, the linked clothes image, the model image and the mask image are input into the first diffusion model, the encoder, the intermediate layer and the decoder of the first diffusion model are frozen, and the pose control vector and the depth control vector are injected into the intermediate layer and the decoder, so that the first diffusion model outputs a try-on image.

[0053] In step S600, the difference loss between the try-on image and the model image is compared, and the first control network and the second control network are trained.

[0054] In this embodiment, the first diffusion model in step S500, the first control network, and the second control network in step S200 work together to form a multi-conditional diffusion model for generating try-on images. The first diffusion model can be, for example, a pre-trained stable diffusion model that belongs to a latent diffusion model. The first diffusion model is frozen, while the first and second control networks are trainable.

[0055] After training, the multi-conditional diffusion model is used to fuse the input target person image (the input path is equivalent to the model image) and clothing image (the target clothing image). It then uses attention to integrate the pose control vector and depth control vector into the denoising Unet network of the first diffusion model. The multi-conditional diffusion model uses a control network architecture based on the pre-trained stable diffusion model, introducing additional control conditions while retaining the generative capabilities of the stable diffusion model. These additional control conditions include pose control vectors and depth control vectors.

[0056] This embodiment uses a model image recording the model's posture and a clothing image of the target garment as input. By extracting a posture feature map and a depth feature map from the model image, the posture feature map and the depth feature map serve as posture control conditions and depth control conditions, respectively, so that the two control networks output posture control variables Co and depth control variables Cd, respectively. Both the posture control variables Co and the depth control variables Cd can provide different and effective conditional information for the first stable diffusion model. This guides the first stable diffusion model to generate natural and realistic fitting images while maintaining the complex features of the clothing images, including text, patterns, and style.

[0057] See also Figure 1 and Figure 2 In some embodiments, optional embodiments and sub-steps of steps S100 to S600 are introduced.

[0058] In step S100, the model image and the mask image are combined to obtain the model latent features, which specifically includes multiplying the model image and the mask image to obtain the model latent features. It can be understood that the mask image is used to retain the portrait information in the model image.

[0059] Step S200, extracting clothing latent features of the clothing image and inputting them into the first control network and the second control network at the same time, specifically comprising: extracting clothing latent features of the clothing image using a feature extractor, and inputting the clothing latent features into the first control network and the second control network simultaneously and separately through the fully connected layer.

[0060] The feature extractor in step S200 can select a computer vision self-supervised model, such as DINO-V2. The feature extractor is used to provide detailed feature information to the model to better control the generation of the try-on image.

[0061] In step S300, the random noise, the model latent feature, and the pose feature map are combined and input into the first control network, specifically including: linking the random noise, the model latent feature, and the mask Figure Three with the pose feature map for addition operation and input into the first control network.

[0062] In step S400, the random noise, the model latent feature, and the depth feature map are combined and input into the second control network, specifically including: linking the random noise, the model latent feature, and the mask Figure Three with the depth feature map for addition operation and input into the second control network.

[0063] Both control networks in step S200 are derived from the first diffusion model in step S400. The first control network includes an encoder and an intermediate layer, both of which are to be trained and derived from the first diffusion model. The second control network includes an encoder and an intermediate layer, both of which are to be trained and derived from the first diffusion model. This embodiment locks all parameters of the first diffusion model, and copies the parameters of the first diffusion model encoder block and the first diffusion model intermediate block (intermediate layer) to the control network, thereby obtaining the first control network and the second control network, respectively.

[0064] In step S500, the first diffusion model outputs the try-on image, which is specifically completed by a VQGAN decoder. The VQGAN decoder is used to map the latent space to the image space to obtain the try-on image.

[0065] Referring to Figure 2 and Figure 3 , in step S500, the intermediate layer and the decoder of the first diffusion model both include frozen self-attention blocks and frozen cross-attention blocks, and a first cross-attention block and a second cross-attention block to be trained are additionally arranged in the feature path. The pose control vector and the depth control vector are injected into the intermediate layer and the decoder, specifically including: injecting the pose control vector into the first cross-attention block, the pose control vector as the first key-value pair for query; and injecting the depth control vector into the second cross-attention block, the depth control vector as the second key-value pair for query.

[0066] Specifically, in order to better integrate the pose control vector and the depth control vector into the frozen pre-trained diffusion model, a multi-control integration module is designed. It includes a trainable first cross-attention block (pose attention block), a trainable second cross-attention block (depth attention block), a frozen self-attention block (belonging to the original first stable model), and a cross-attention block (belonging to the original first stable model) arranged along the feature path.

[0067] As Figure 3As shown, the multi-control injection module enhances the original attention modules (including the original self-attention module and the original criss-cross attention module) in the first diffusion model denoising U-Net by introducing trainable pose attention and deep attention, while keeping the self-attention and criss-cross attention components frozen. The benefit of this structure is that it can enhance all attention modules applied to the intermediate and decoder of the denoising U-Net. Specifically, for pose attention, the query vector Q is derived from the latent features of the first diffusion model denoising U-Net, and the key-value pairs (key K and value V) are derived from the pose control vector. For deep attention, the query vector Q is derived from the latent features of the first diffusion model denoising U-Net, and the key-value pairs (key K and value V) are derived from the deep control vector. For frozen criss-cross attention, this project uses CLIP embedding (Contrastive Language-Image Pre-training) as the key K and value V. CLIP embedding is specifically obtained by inputting the clothing image into the CLIP image encoder.

[0068] During training, this project specifically performs gradient updates on the parameters of pose attention and depth attention in the control network and multi-control injection module.

[0069]

[0070] Where,

[0071] ∈, represents the real noise;

[0072] ∈ θ , represents the prediction noise at time step t;

[0073] x t , represents random noise;

[0074] F(x m ), represents the model latent features, F represents feature extraction, x m Represents the model image multiplied by the mask image;

[0075] m, represents the mask image;

[0076] g, represents clothing image;

[0077] d, represents the depth control vector, i.e. Cd;

[0078] o, represents the posture control vector, i.e. Co;

[0079] t represents the time step.

[0080] Steps S100 to S600 generate natural, complex-featured try-on images using a multi-conditional control model. However, the resolution of these images often fails to meet expectations. Current virtual try-on methods can only generate try-on images with a maximum resolution of 1K, but actual application scenarios may require higher resolution images, such as 2K. The most direct approach is to upsample the generated try-on images using a natural image stabilization diffusion model. However, directly using existing super-resolution methods may result in distortion in the upsampled try-on images.

[0081] See also Figure 4 and Figure 5 Based on this, in some embodiments, a super-resolution diffusion model for virtual try-on is provided, and a virtual try-on method based on the diffusion model is specifically described in steps S700 to S900, for upsampling a low-resolution try-on image to a high-resolution image while maintaining the original texture features. Among them:

[0082] Step S700 : extracting latent features of the try-on image, upsampling the try-on image to obtain a sampling image of expected resolution, and extracting latent features of the sampling image.

[0083] In step S800 , random noise, latent features of the try-on image, and latent features of the sampling image are input into a third control network, and the third control network outputs a control vector.

[0084] In step S900, random noise is input into the frozen second diffusion model, and a control vector is injected into the second diffusion model, thereby guiding the second diffusion model to output a fitting image of expected resolution, obtaining a sample image of expected resolution corresponding to the model image, comparing the sample image of expected resolution with the fitting image of expected resolution, and the difference loss between the two, and optimizing the third control network.

[0085] In steps S700 to S900: The try-on image (or low-resolution try-on image) of step S600 is upsampled to obtain a sampling image of the expected resolution. At this time, the sampling image obtained lacks image details. The latent features of the try-on image are extracted by DINO-V2, and encoded into the latent space of the third control network through the fully connected layer to obtain the control vector output by the third control network. This project uses DINO-V2 as a feature extractor for low-resolution images in the control network. Compared with CLIP extraction, DINO-V2 encodes the image not only as a global tag, but also as a patch token, which helps to retain the information of the low-resolution image and provide a detailed representation.

[0086] In step S800, a third control network includes an encoder and an intermediate layer, both of which are to be trained and are derived from a second diffusion model. The second diffusion model can be a pre-trained, stable natural image diffusion model. The second diffusion model combined with the third control network is a super-resolution diffusion model.

[0087] This embodiment utilizes latent features of the try-on image and the sampled image based on the try-on image, and provides prior knowledge about the model and clothing through a third control network, making the second diffusion model pixel-controllable. The control vector input is derived from the try-on image, which can be used to supervise the generation of the details of the try-on image. This ensures that the content details of the try-on image are preserved during the process of reconstructing the try-on image into a high-resolution image.

[0088] In step S900, the control vector is injected into the second diffusion model. Specifically, the second diffusion model includes an encoder, an intermediate layer, and a decoder. The intermediate layer and the decoder each include a frozen self-attention block and a frozen cross-attention block. The control vector is input to the cross-attention block. The control vector serves as a third key-value pair (key K and value V) for query, replacing the original CLIP embedding. Step S900 uses cross-attention to inject the control vector into the frozen Unet network of the second diffusion model. See the following formula for details:

[0089]

[0090] Where,

[0091] ∈, represents the real noise;

[0092] ∈ θ , represents the prediction noise;

[0093] z t , represents random noise;

[0094] F(LR) represents the latent features of the sampled image extracted (using the VQGAN encoder);

[0095] c, represents the control vector;

[0096] t represents the time step.

[0097] During inference, the generated low-resolution try-on images are input as conditions into the fine-tuned diffusion model. After t iterations, high-fidelity high-resolution try-on images containing clothing detail features are obtained.

[0098] See also Figure 6The various embodiments of this application provide a cascaded diffusion model that improves the controllability of virtual try-on image generation while generating high-resolution try-on images. The cascaded diffusion model includes a multi-conditional diffusion model and a super-resolution diffusion model. The former is used to generate try-on images with complex feature details, while the latter is used to improve the resolution of try-on images while preserving image details.

[0099] Specifically, a multi-conditional diffusion model is used to generate low-resolution fitting images. By injecting pose and depth control vectors into a denoising U-net, the resulting fitting images retain the complex features of the garment. The super-resolution diffusion model primarily consists of a frozen first diffusion model and a trainable third control network, which is used to upsample the generated low-resolution results to high resolution. The various embodiments of this application can integrate a given garment image onto a target person, generating realistic, high-resolution fitting images.

[0100] It should be understood that although Figure 1 and Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 and Figure 4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0101] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a virtual try-on method based on a diffusion model is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0102] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0103] Step S100: Obtain a clothing image and an image of a model wearing the corresponding clothing, obtain a posture image, a depth image, and a mask image based on the model image, and obtain the model's latent features by combining the model image and the mask image.

[0104] Step S200 , extracting clothing latent features from the clothing image and simultaneously inputting them into the first control network and the second control network.

[0105] In step S300, the posture graph and the clothing graph are convolved to obtain a posture feature graph, random noise, model latent features, and the posture feature graph are combined and input into a first control network, and the first control network outputs a posture control vector.

[0106] In step S400, the depth map and the clothing map are convolved to obtain a depth feature map, random noise, model latent features, and the depth feature map are combined and input into the second control network, and the second control network outputs a depth control vector.

[0107] Step S500: Provide a first diffusion model, input the linked clothing image, model image and mask image into the first diffusion model, freeze the encoder, intermediate layer and decoder of the first diffusion model, and inject the posture control vector and depth control vector into the intermediate layer and decoder, so that the first diffusion model outputs a try-on image.

[0108] Step S600 , comparing the difference loss between the try-on image and the model image, and training the first control network and the second control network.

[0109] In one embodiment, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the following steps:

[0110] Step S100: Obtain a clothing image and an image of a model wearing the corresponding clothing, obtain a posture image, a depth image, and a mask image based on the model image, and obtain the model's latent features by combining the model image and the mask image.

[0111] Step S200 , extracting clothing latent features from the clothing image and simultaneously inputting them into the first control network and the second control network.

[0112] In step S300, the posture graph and the clothing graph are convolved to obtain a posture feature graph, random noise, model latent features, and the posture feature graph are combined and input into a first control network, and the first control network outputs a posture control vector.

[0113] In step S400, the depth map and the clothing map are convolved to obtain a depth feature map, random noise, model latent features, and the depth feature map are combined and input into the second control network, and the second control network outputs a depth control vector.

[0114] Step S500: Provide a first diffusion model, input the linked clothing image, model image and mask image into the first diffusion model, freeze the encoder, intermediate layer and decoder of the first diffusion model, and inject the posture control vector and depth control vector into the intermediate layer and decoder, so that the first diffusion model outputs a try-on image.

[0115] Step S600 , comparing the difference loss between the try-on image and the model image, and training the first control network and the second control network.

[0116] In this embodiment, the computer program product includes a program code portion for executing the steps of the diffusion model-based virtual try-on method in each embodiment of the present application when the computer program product is executed by one or more computing devices. The computer program product can be stored on a computer-readable recording medium. The computer program product can also be provided for download via a data network (e.g., via a RAN, via the Internet, and / or via an RBS). Alternatively or additionally, the method can be encoded in a field programmable gate array (FPGA) and / or an application-specific integrated circuit (ASIC), or the functionality can be provided for download with the aid of a hardware description language.

[0117] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0118] The technical features of the above embodiments may be combined in any manner. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there are no conflicts in the combination of these technical features, they should be considered to be within the scope of this specification. When technical features in different embodiments are reflected in the same figure, it can be regarded as that figure also discloses the combination examples of the various embodiments involved.

[0119] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A virtual try-on method based on a diffusion model, characterized in that: The virtual try-on method is used to integrate a given clothing image into a target person image to generate a try-on image, and includes: Obtaining a clothing image and an image of a model wearing the corresponding clothing, obtaining a posture image, a depth image, and a mask image based on the model image, respectively, and combining the model image and the mask image to obtain model latent features; extracting the clothing latent features from the clothing image and simultaneously inputting them into a first control network and a second control network; convolving the posture image and the clothing image to obtain a posture feature map, combining random noise, the model latent features, and the posture feature map, and inputting them into the first control network, which outputs a posture control vector; Convolving the depth map with the clothing image to obtain a depth feature map, combining random noise, the model latent features, and the depth feature map and inputting them into the second control network, the second control network outputting a depth control vector; providing a first diffusion model, inputting the linked clothing image, the model image, and the mask image into the first diffusion model, freezing the encoder, intermediate layer, and decoder of the first diffusion model, and injecting the posture control vector and the depth control vector into the intermediate layer and decoder, so that the first diffusion model outputs a try-on image; comparing the difference loss between the try-on image and the model image, and training the first control network and the second control network; The latent features of the try-on image are extracted, the try-on image is upsampled to obtain a sampling image of expected resolution, and the latent features of the sampling image are extracted from the sampling image; random noise, the latent features of the try-on image, and the latent features of the sampling image are input into a third control network, and the third control network outputs a control vector; random noise is input into a frozen second diffusion model, and the control vector is injected into the second diffusion model, thereby guiding the second diffusion model to output a try-on image of expected resolution; a sample image of expected resolution corresponding to the model image is obtained, the sample image of expected resolution and the try-on image of expected resolution are compared, and the difference loss between the two is compared, and the third control network is optimized.

2. The virtual try-on method based on the diffusion model according to claim 1, characterized in that: Extracting clothing latent features from the clothing image and simultaneously inputting them into the first control network and the second control network specifically includes: A feature extractor is used to extract clothing latent features of the clothing image, and the clothing latent features are simultaneously input into the first control network and the second control network through a fully connected layer.

3. The virtual try-on method based on the diffusion model according to claim 1, characterized in that: Combining random noise, the model latent features, and the posture feature map and inputting the combined results into the first control network specifically includes: After linking the random noise, the model latent features and the mask map, the three are summed with the posture feature map and input into the first control network; Combining random noise, the model latent features, and the depth feature map and inputting the combined results into the second control network specifically includes: After linking the random noise, the model latent features and the mask map, the three are added together with the depth feature map and input into the second control network.

4. The virtual try-on method based on the diffusion model according to claim 1, characterized in that: The first control network includes an encoder and an intermediate layer, both to be trained and both derived from a first diffusion model; The second control network includes an encoder and an intermediate layer, both to be trained and both derived from the first diffusion model.

5. The virtual try-on method based on the diffusion model according to claim 1, characterized in that: The intermediate layer and the decoder of the first diffusion model both include frozen self-attention blocks and frozen cross-attention blocks, and a first cross-attention block and a second cross-attention block to be trained are added in the feature path; Injecting the posture control vector and the depth control vector into the intermediate layer and the decoder specifically includes: Injecting the posture control vector into the first cross-attention block, the posture control vector serving as a first key-value pair for query; The depth control vector is injected into the second cross-attention block, and the depth control vector serves as a second key-value pair for query.

6. The virtual try-on method based on the diffusion model according to claim 1, characterized in that: The third control network includes an encoder and an intermediate layer, both to be trained and both derived from the second diffusion model.

7. The virtual try-on method based on the diffusion model according to claim 6, characterized in that: Injecting the control vector into the second diffusion model specifically includes: The second diffusion model includes an encoder, an intermediate layer, and a decoder, wherein the intermediate layer and the decoder each include a self-attention block and a cross-attention block; The control vector is input to the cross-attention block, and the control vector serves as a third key-value pair for query.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that The processor executes the computer program to implement the steps of the virtual try-on method based on the diffusion model according to any one of claims 1 to 7.

9. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the virtual try-on method based on the diffusion model as claimed in any one of claims 1 to 7 are implemented.

Images