Virtual clothing changing method, device, equipment, and storage medium

Abstract

The present application provides a virtual clothing-changing method, device, equipment, and storage medium, which conveniently includes: obtaining a person image and a clothing image; generating a target image based on a diffusion model; the person image in the target image wears a clothing image; wherein the diffusion model includes a forward process and a reverse process; the forward process generates a noisy image of the target image; in the reverse process, the Transformer block has no cross-attention layer, and each Transformer block performs variational encoder embedding and CLIP visual embedding with the clothing image. In the reverse process of the diffusion model provided by the present application, the Transformer block has no cross-attention layer, and each Transformer block performs variational encoder embedding and CLIP visual embedding with the clothing image, which realizes clothing-centric visual adaptation of the denoising process and improves the consistency of the details of the clothes and the posture of the person in the target image.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a virtual clothing changing method, device, equipment, and storage medium. Background Art

[0002] Image-based virtual try-on task is an important research topic in the field of computer vision, and a practical virtual try-on system has great potential in real-world applications, such as online shopping. This technology aims to synthesize images of a specific person wearing the desired clothing.

[0003] The basic methods of image-based virtual try-on tasks can be divided into two categories: methods based on GAN (Generative adversarial network) generative models and methods based on diffusion models. The first method uses GAN to synthesize the final character image based on distorted clothing and human body conditions. Most of these works can be divided into two stages. The first stage is to use a clothing distortion model to distort the clothing to the posture of the target model. The second stage uses the GAN knot and distorted clothing and model images to generate the final clothing changing effect. The second method is to introduce a diffusion model into the process of the virtual clothing changing task, and directly inject the deformed clothing obtained from the distortion module into the diffusion model to achieve virtual clothing changing.

[0004] Existing image-based virtual try-on methods are limited by the ability of GAN itself to handle complex human poses. Or because during the diffusion process in the latent space, the compressed latent features cannot remember every complex clothing detail, making it impossible for the final generated results to fully retain every detail of the clothing in the store, especially the high-frequency texture details of complex patterns or text. Summary of the invention

[0005] In order to solve one of the above-mentioned technical defects, the present application provides a virtual clothing changing method, device, equipment, and storage medium.

[0006] In a first aspect of the present application, a virtual clothing changing method is provided, the method comprising:

[0007] Obtain person images and clothing images;

[0008] Generate a target image based on a diffusion model; a person image in the target image is wearing a clothing image;

[0009] Among them, the diffusion model includes the forward process and the reverse process;

[0010] The forward process generates a noisy image of the target image;

[0011] In the reverse process, the Transformer block has no cross-attention layer, and each Transformer block uses the clothing image for variational encoder embedding and CLIP visual embedding.

[0012] Optionally, the inverse process of the diffusion model includes:

[0013] Deform the clothing image to obtain a deformed image;

[0014] Extract an irrelevant image that has nothing to do with clothing from the person image;

[0015] The noisy image is concatenated with the deformed image and the irrelevant image in the channel dimension;

[0016] The stitched image is used as input to the inverse process.

[0017] Optionally, the loss function used by the denoising network during the inverse of the diffusion model is

[0018] in, is the mean square error loss, is the appearance loss, and λ is a hyperparameter used to balance the mean square error loss and appearance loss;

[0019] ∈ is noise, ∈ θ (γ,I c ,I w ,t) predicted noise, γ is the spliced ​​image, I c is a clothing image, I w is the deformed image, t is the time step, and θ is the prediction parameter;

[0020] is the spatial perception loss, Enhance loss for high frequencies;

[0021] is a function that captures the difference in structure and texture, ⊙ is a multiplication operator, is the predicted target image, I is the target image, m w is the deformation mask, m w It is obtained by deforming the clothing image;

[0022] For The edge image extracted from h is the edge image extracted from I.

[0023] Optionally, the variational encoder embeds f vae =MLP vae (ε(I w));

[0024] Among them, MLP vae (·) is the multilayer perceptron embedded in the variational encoder, ε(·) is the variational encoder, I w is a deformed image, the I w It is obtained by deforming the clothing image.

[0025] Optionally, CLIP visual embedding f clip =MLP clip (CLIP v (I c ));

[0026] Among them, MLP clip (·) is the multi-layer perceptron of CLIP visual embedding, CLIP v (·) is the CLIP visual encoder, I c For clothing images.

[0027] Optionally, the criss-cross attention mechanism in the reverse process is:

[0028]

[0029] Among them, Z attn is the output of the cross attention mechanism, Softmax(·) is the normalization function, Q is the query in the cross attention, K clip is the key of CLIP, T is the transpose, d is the dimension of the key, V clip is the value of CLIP, K vae is the key of the variational encoder, V vae is the value of the variational encoder;

[0030] Q=f q W q , f q is the feature of the previous self-attention layer, W q is the query projection matrix in cross attention;

[0031] f clip is the CLIP feature, is the key projection matrix of CLIP;

[0032] The projection matrix for the CLIP value;

[0033] f vae is the variational encoder feature, is the key projection matrix of the variational encoder;

[0034] is the value projection matrix of the variational encoder.

[0035] Optionally, the criss-cross attention mechanism in the reverse process is:

[0036]

[0037] Among them, Softmax(·) is the normalization function, Q is the query in the cross attention, and K clip is the key of CLIP, is the transpose, d is the dimension of the key, V clip is the value of CLIP, K vae is the key of the variational encoder, V vae is the value of the variational encoder, ⊙ is the multiplication operator, M attn is the attention mask obtained based on the deformation mask, which is obtained by deforming the clothing image;

[0038] Q=f q W q , f q is the feature of the previous self-attention layer, W q is the query projection matrix in cross attention;

[0039] f clip is the CLIP feature, is the key projection matrix of CLIP;

[0040] The projection matrix for the CLIP value;

[0041] f vae is the variational encoder feature, is the key projection matrix of the variational encoder;

[0042] is the value projection matrix of the variational encoder.

[0043] In a second aspect of the present application, a virtual clothing changing device is provided, the device comprising:

[0044] An acquisition module, used for acquiring character images and clothing images;

[0045] A generation module is used to generate a target image based on a diffusion model; a person image in the target image is wearing a clothing image;

[0046] Among them, the diffusion model includes the forward process and the reverse process;

[0047] The forward process generates a noisy image of the target image;

[0048] In the reverse process, the Transformer block has no cross-attention layer, and each Transformer block uses the clothing image for variational encoder embedding and CLIP visual embedding.

[0049] In a third aspect of the present application, an electronic device is provided, including:

[0050] Memory;

[0051] Processor; and

[0052] Computer programs;

[0053] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect above.

[0054] In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored; the computer program is executed by a processor to implement the method described in the first aspect above.

[0055] The present application provides a virtual clothing-changing method, device, equipment, and storage medium, which conveniently includes: obtaining a person image and a clothing image; generating a target image based on a diffusion model; the person image in the target image wears a clothing image; wherein the diffusion model includes a forward process and a reverse process; the forward process generates a noisy image of the target image; in the reverse process, the Transformer block has no cross-attention layer, and each Transformer block performs variational encoder embedding and CLIP visual embedding with the clothing image. In the reverse process of the diffusion model provided by the present application, the Transformer block has no cross-attention layer, and each Transformer block performs variational encoder embedding and CLIP visual embedding with the clothing image, which realizes clothing-centric visual adaptation of the denoising process and improves the consistency of the details of the clothes and the posture of the person in the target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0057] Figure 1 A flowchart of a virtual clothing changing method provided in an embodiment of the present application;

[0058] Figure 2 A schematic diagram of an implementation architecture of a virtual clothing changing method provided in an embodiment of the present application;

[0059] Figure 3A schematic diagram of the structure of a virtual clothing changing device provided in an embodiment of the present application;

[0060] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the technical solutions and advantages in the embodiments of the present application more clearly understood, the exemplary embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0062] In the process of implementing this application, the inventors found that the existing image-based virtual try-on methods are limited by the ability of GAN itself to handle complex human poses. Or because the compressed latent features cannot remember every complex clothing detail during the diffusion process in the latent space, the final generated result cannot fully retain every detail of the clothing in the store, especially the high-frequency texture details of complex patterns or text.

[0063] In view of the above problems, the present application provides a virtual clothing-changing method, device, equipment, and storage medium, which conveniently includes: obtaining a person image and a clothing image; generating a target image based on a diffusion model; the person image in the target image wears a clothing image; wherein the diffusion model includes a forward process and a reverse process; the forward process generates a noisy image of the target image; in the reverse process, the Transformer block has no cross-attention layer, and each Transformer block performs variational encoder embedding and CLIP visual embedding with the clothing image. In the reverse process of the diffusion model provided by the present application, the Transformer block has no cross-attention layer, and each Transformer block performs variational encoder embedding and CLIP visual embedding with the clothing image, thereby realizing clothing-centric visual adaptation of the denoising process and improving the consistency of the details of the clothes and the person's posture in the target image.

[0064] See also Figure 1 The implementation process of the virtual clothing changing method provided in this embodiment is as follows:

[0065] 101, obtaining a person image and a clothing image.

[0066] 102, generating a target image based on a diffusion model. In the target image, a person image is wearing a clothing image.

[0067] Among them, the diffusion model includes forward process and reverse process.

[0068] The forward process generates a noisy image of the target image.

[0069] In the reverse process, the Transformer block has no cross-attention layer, and each Transformer block uses the clothing image for variational encoder embedding and CLIP visual embedding.

[0070] The diffusion model used in this embodiment is different from the existing diffusion model only in the Transformer block. This embodiment replaces the cross attention layer in each Transformer block with a clothing-centric visual adapter. In other words, the Transformer block in this embodiment has no cross attention layer, but a visual adapter. The visual adapter performs variational encoder embedding and CLIP visual embedding with clothing images. The visual adapter can effectively use clothing images to generate high-fidelity character images (i.e., target images).

[0071] In specific implementation, the method provided in this embodiment can be based on Figure 2 In the virtual dressing task, a character image of a user is obtained through step 101. and an image of clothing in a store In step 102, a high-quality realistic person image (i.e., target image) is synthesized. Among them, the person in the target image wears the target clothing.

[0072] In the forward process of the diffusion model, a noisy image of the target image is generated. For example, noise is gradually added to the target image I according to the Markov chain. Specifically, the target image I can be mapped to the latent space using VAE (Variational Autoencoder) ε(·) to obtain the mapped image x0 = ε(I). Then, at any time step t∈[1,1000], noise ∈ is added to x0 to obtain the noisy image at any time step t Among them, α t is the noise control parameter at time step t.

[0073] In the inverse process of the diffusion model, the following steps are performed:

[0074] 201, deforming the clothing image to obtain a deformed image.

[0075] When the deformed image is obtained, a deformation mask is also obtained. That is, in step 201, the clothing image (i.e., the clothing to be changed) is deformed. c Deformation is performed to maximize the match between the target garment and the body posture, resulting in a deformed image (i.e., deformed garment) and deformation mask m w ∈{0,1} H×W .

[0076] Deformation mask m w Represents the deformed image (i.e., deformed clothing) I w area.

[0077] 202, extracting an irrelevant image irrelevant to clothing from the person image.

[0078] In order to retain the area not related to clothing, in step 202, the image I p A clothing-irrelevant image (i.e., irrelevant image) is extracted from a , I a The area of ​​the garment is completely covered.

[0079] 203, concatenate the noisy image, the deformed image, and the irrelevant image in the channel dimension.

[0080] When stitching, we can first use ε(·) to map the deformed image and the irrelevant image to the latent space, and then stitch them in the channel dimension. w Mapping to the latent space to get x w , that is, x w =ε(I w ). Use ε(·) to convert the irrelevant image I a Mapping to the latent space to get x a , that is, x w =ε(I a ).

[0081] The concatenation process is: γ = Concat(x w ,x a ,x t ), where γ is the spliced ​​image, x t is the noisy image, and Concat(·) is the concatenation function.

[0082] At 204, the stitched image is used as input to the reverse process.

[0083] That is, the spliced ​​image γ is used as the input of the denoising network (UNet) in the reverse process of the diffusion model.

[0084] In the reverse process of the diffusion model, the denoising network (UNet) is trained based on the clothing image I c and the deformed image I w Predict the added noise ∈.

[0085] Loss function used by the denoising network (UNet)

[0086] in, is the mean square error loss, is the appearance loss, and λ is a hyperparameter used to balance the mean square error loss and appearance loss.

[0087] ∈ is noise, ∈ θ (γ,I c ,I w ,t) predicted noise, γ is the spliced ​​image, I c is a clothing image, I w is the deformed image, t is the time step, and θ is the prediction parameter.

[0088] Through the appearance loss L appearance It can strengthen the consistency of high-frequency details of synthetic clothing and clothing images, thus achieving higher fidelity and fine texture. As a composite adaptive loss, appearance loss Can be decomposed into two components: spatial perception loss and high frequency enhancement loss Right now

[0089] is a function that captures the difference in structure and texture, ⊙ is a multiplication operator, is the predicted target image, I is the target image, m w is the deformation mask, m w It is obtained by deforming the clothing image.

[0090] For The edge image extracted from h is the edge image extracted from I.

[0091] Through space-aware loss The structural and texture differences between the predicted and real images can be captured in the perceptual feature space, rather than just at the pixel level.

[0092] By enhancing the loss of high frequency The model's ability to generate high-frequency details can be enhanced. The visual adapter of this embodiment uses edge detection to extract high-frequency information. For example, the horizontal and vertical Sobel kernels are used as high-pass filters to extract high-frequency information from the predicted target image. Extract edge images Extract edge image I from target image I h .

[0093] The reverse process implements clothing-centric adaptation, which is implemented by the visual adapter. In this embodiment, the cross-attention layer in each Transformer block is replaced by the visual adapter. The visual adapter takes two appearance priors as input: 1) the clothing image I c A variant version of Iw VAE (Variational Encoder) embedding is used to recover complex patterns in synthetic images. 2) Clothing Image I c CLIP (Contrastive Language-Image Pre-Training) visual embedding is used to generate the overall structure without considering the deformation results. w imperfections.

[0094] For example, the VAE (Variational Encoder) embeds f vae =MLP vae (ε(I w )).

[0095] Among them, MLP vae (·) is the multilayer perceptron embedded in the variational encoder, ε(·) is the variational encoder, I w is a deformed image, the I w The clothing image is deformed.

[0096] CLIP Visual Embedding clip =MLP clip (CLIP v (I c )).

[0097] Among them, MLP clip (·) is the multi-layer perceptron of CLIP visual embedding, CLIP v (·) is the CLIP visual encoder, I c For clothing images.

[0098] In addition, considering this vae and f clip Focusing on different granularity levels of clothing images, the cross-attention mechanism adopted by the visual adapter adopted in this embodiment is a decoupled cross-attention mechanism.

[0099] For example, the cross-attention mechanism used by the visual adapter in the reverse process is:

[0100]

[0101] Among them, Softmax(·) is the normalization function, Q is the query in the cross attention, and K clip is the key of CLIP, T is the transpose, d is the dimension of the key, V clip is the value of CLIP, K vae is the key of the variational encoder, V vae is the value of the variational encoder.

[0102] Q=f q Wq , f q is the feature of the previous self-attention layer, W q is the query projection matrix in the cross attention.

[0103] f clip is the CLIP feature, The key projection matrix for CLIP.

[0104] The projection matrix for the CLIP value.

[0105] f vae is the variational encoder feature, is the key projection matrix of the variational encoder.

[0106] is the value projection matrix of the variational encoder.

[0107] If you want to achieve local fine-grained alignment of the reference clothing and human posture, you can also use the deformation mask m w Downsampled to an attention mask M that matches the corresponding attention layer in the vision adapter attn , which is used to suppress the attention weights that are not related to the clothing area in the attention map. At this time, the cross attention mechanism adopted by the visual adapter in the reverse process is:

[0108]

[0109] Among them, ⊙ is the multiplication operator, M attn is the attention mask obtained based on the deformation mask, where the deformation mask is obtained by deforming the clothing image.

[0110] The virtual clothing changing method provided in this embodiment upgrades the diffusion model for the virtual clothing changing task by enhancing the visual appearance and the guidance of high-frequency texture details. The diffusion model adopted in this embodiment gradually mines more prior knowledge about fine-grained clothing details to improve the consistency of clothing details and character postures in the virtual try-on results.

[0111] The virtual clothing-changing method provided in this embodiment adopts a clothing-centric virtual clothing-changing diffusion model, which can preserve the details of the target clothing in the virtual try-on task. The appearance priors of the reference clothing are reshaped by the CLIP visual encoder and the VAE encoder, and these priors are integrated into the UNet through the clothing-focused visual adapter. In this way, the diffusion process is effectively strengthened by the enhanced appearance guidance of the given clothing. In addition, through the appearance loss, it can be ensured that the synthesized clothing is consistent with the target clothing in high-frequency details and geometry.

[0112] This embodiment provides a virtual clothing-changing method, which obtains a person image and a clothing image; generates a target image based on a diffusion model; the person image in the target image wears a clothing image; wherein the diffusion model includes a forward process and a reverse process; the forward process generates a noisy image of the target image; in the reverse process, the Transformer block has no cross-attention layer, and each Transformer block performs variational encoder embedding and CLIP visual embedding with the clothing image. In the reverse process of the diffusion model provided by the present application, the Transformer block has no cross-attention layer, and each Transformer block performs variational encoder embedding and CLIP visual embedding with the clothing image, which realizes clothing-centric visual adaptation in the denoising process and improves the consistency of clothing details and person postures in the target image.

[0113] Based on the same inventive concept of the virtual clothing changing method, this embodiment provides a virtual clothing changing device, see Figure 3 , the device comprises:

[0114] The acquisition module 301 is used to acquire a person image and a clothing image.

[0115] The generating module 302 is used to generate a target image based on the diffusion model. The target image includes a person image wearing a clothing image.

[0116] Among them, the diffusion model includes forward process and reverse process.

[0117] The forward process generates a noisy image of the target image.

[0118] In the reverse process, the Transformer block has no cross-attention layer, and each Transformer block uses the clothing image for variational encoder embedding and CLIP visual embedding.

[0119] The inverse process of the diffusion model includes:

[0120] The clothing image is deformed to obtain a deformed image.

[0121] Extract an irrelevant image from the person image that has nothing to do with clothing.

[0122] The noisy image is concatenated with the deformed image and the irrelevant image in the channel dimension.

[0123] The stitched image is used as input to the inverse process.

[0124] Among them, in the reverse process of the diffusion model, the loss function used by the denoising network is

[0125] in, is the mean square error loss, is the appearance loss, and λ is a hyperparameter used to balance the mean square error loss and appearance loss.

[0126] ∈ is noise, ∈ θ (γ,I c ,I w ,t) predicted noise, γ is the spliced ​​image, I c is a clothing image, I w is the deformed image, t is the time step, and θ is the prediction parameter.

[0127] is the spatial perception loss, It is the high frequency enhancement loss.

[0128] is a function that captures the difference in structure and texture, ⊙ is a multiplication operator, is the predicted target image, I is the target image, m w is the deformation mask, m w It is obtained by deforming the clothing image.

[0129] For The edge image extracted from h is the edge image extracted from I.

[0130] Among them, the variational encoder embeds f vae =MLP vae (ε(I w )).

[0131] Among them, MLP vae (·) is the multilayer perceptron embedded in the variational encoder, ε(·) is the variational encoder, I w is a deformed image, the I w It is obtained by deforming the clothing image.

[0132] Among them, CLIP visual embedding f clip =MLP clip (CLIP v (I c )).

[0133] Among them, MLP clip (·) is the multi-layer perceptron of CLIP visual embedding, CLIP v (·) is the CLIP visual encoder, I c For clothing images.

[0134] Optionally, the criss-cross attention mechanism in the reverse process is:

[0135]

[0136] Among them, Z attn is the output of the cross attention mechanism, Softmax(·) is the normalization function, Q is the query in the cross attention, K clip is the key of CLIP, T is the transpose, d is the dimension of the key, V clip is the value of CLIP, K vae is the key of the variational encoder, V vae is the value of the variational encoder.

[0137] Q=f q W q , f q is the feature of the previous self-attention layer, W q is the query projection matrix in the cross attention.

[0138] f clip is the CLIP feature, The key projection matrix for CLIP.

[0139] The projection matrix for the CLIP value.

[0140] f vae is the variational encoder feature, is the key projection matrix of the variational encoder.

[0141] is the value projection matrix of the variational encoder.

[0142] Among them, the cross attention mechanism in the reverse process is:

[0143]

[0144] Among them, Softmax(·) is the normalization function, Q is the query in the cross attention, and K clip is the key of CLIP, is the transpose, d is the dimension of the key, V clip is the value of CLIP, K vae is the key of the variational encoder, V vae is the value of the variational encoder, ⊙ is the multiplication operator, M attn is the attention mask obtained based on the deformation mask, where the deformation mask is obtained by deforming the clothing image.

[0145] Q=f q W q , f q is the feature of the previous self-attention layer, W qis the query projection matrix in the cross attention.

[0146] f clip is the CLIP feature, The key projection matrix for CLIP.

[0147] The projection matrix for the CLIP value.

[0148] f vae is the variational encoder feature, is the key projection matrix of the variational encoder.

[0149] is the value projection matrix of the variational encoder.

[0150] The device provided in this embodiment obtains a person image and a clothing image; generates a target image based on a diffusion model; the person image in the target image wears a clothing image; wherein the diffusion model includes a forward process and a reverse process; the forward process generates a noisy image of the target image; in the reverse process, the Transformer block has no cross-attention layer, and each Transformer block performs variational encoder embedding and CLIP visual embedding with the clothing image. In the reverse process of the diffusion model provided in the present application, the Transformer block has no cross-attention layer, and each Transformer block performs variational encoder embedding and CLIP visual embedding with the clothing image, thereby achieving clothing-centric visual adaptation of the denoising process and improving the consistency of clothing details and person postures in the target image.

[0151] Based on the same inventive concept of the virtual clothing changing method, this embodiment provides an electronic device, such as Figure 4 As shown, it includes: a memory 401, a processor 402, and a computer program.

[0152] The computer program is stored in the memory 401 and is configured to be executed by the processor 402 to implement the above-mentioned virtual clothing changing method.

[0153] Specifically,

[0154] Get person images and clothing images.

[0155] Generate a target image based on the diffusion model. The target image contains a person image wearing a clothing image.

[0156] Among them, the diffusion model includes forward process and reverse process.

[0157] The forward process generates a noisy image of the target image.

[0158] In the reverse process, the Transformer block has no cross-attention layer, and each Transformer block uses the clothing image for variational encoder embedding and CLIP visual embedding.

[0159] Optionally, the inverse process of the diffusion model includes:

[0160] The clothing image is deformed to obtain a deformed image.

[0161] Extract an irrelevant image from the person image that has nothing to do with clothing.

[0162] The noisy image is concatenated with the deformed image and the irrelevant image in the channel dimension.

[0163] The stitched image is used as input to the inverse process.

[0164] Optionally, the loss function used by the denoising network during the inverse of the diffusion model is

[0165] in, is the mean square error loss, is the appearance loss, and λ is a hyperparameter used to balance the mean square error loss and appearance loss.

[0166] ∈ is noise, ∈ θ (γ,I c ,I w ,t) predicted noise, γ is the spliced ​​image, I c is a clothing image, I w is the deformed image, t is the time step, and θ is the prediction parameter.

[0167] is the spatial perception loss, It is the high frequency enhancement loss.

[0168] is a function that captures the difference in structure and texture, ⊙ is a multiplication operator, is the predicted target image, I is the target image, m w is the deformation mask, m w It is obtained by deforming the clothing image.

[0169] For The edge image extracted from h is the edge image extracted from I.

[0170] Optionally, the variational encoder embeds f vae =MLP vae (ε(I w)).

[0171] Among them, MLP vae (·) is the multilayer perceptron embedded in the variational encoder, ε(·) is the variational encoder, I w is a deformed image, the I w It is obtained by deforming the clothing image.

[0172] Optionally, CLIP visual embedding f clip =MLP clip (CLIP v (I c )).

[0173] Among them, MLP clip (·) is the multi-layer perceptron of CLIP visual embedding, CLIP v (·) is the CLIP visual encoder, I c For clothing images.

[0174] Optionally, the criss-cross attention mechanism in the reverse process is:

[0175]

[0176] Among them, Z attn is the output of the cross attention mechanism, Softmax(·) is the normalization function, Q is the query in the cross attention, K clip is the key of CLIP, T is the transpose, d is the dimension of the key, V clip is the value of CLIP, K vae is the key of the variational encoder, V vae is the value of the variational encoder.

[0177] Q=f q W q , f q is the feature of the previous self-attention layer, W q is the query projection matrix in the cross attention.

[0178] f clip is the CLIP feature, The key projection matrix for CLIP.

[0179] The projection matrix for the CLIP value.

[0180] f vae is the variational encoder feature, is the key projection matrix of the variational encoder.

[0181] is the value projection matrix of the variational encoder.

[0182] Optionally, the criss-cross attention mechanism in the reverse process is:

[0183]

[0184] Among them, Softmax(·) is the normalization function, Q is the query in the cross attention, and K clip is the key of CLIP, T is the transpose, d is the dimension of the key, V clip is the value of CLIP, K vae is the key of the variational encoder, V vae is the value of the variational encoder, ⊙ is the multiplication operator, M attn is the attention mask obtained based on the deformation mask, where the deformation mask is obtained by deforming the clothing image.

[0185] Q=f q W q , f q is the feature of the previous self-attention layer, W q is the query projection matrix in the cross attention.

[0186] f clip is the CLIP feature, The key projection matrix for CLIP.

[0187] The projection matrix for the CLIP value.

[0188] f vae is the variational encoder feature, is the key projection matrix of the variational encoder.

[0189] is the value projection matrix of the variational encoder.

[0190] The electronic device provided by this embodiment has a computer program executed by a processor to obtain a person image and a clothing image; a target image is generated based on a diffusion model; the person image in the target image wears a clothing image; wherein the diffusion model includes a forward process and a reverse process; the forward process generates a noisy image of the target image; in the reverse process, the Transformer block has no cross-attention layer, and each Transformer block performs variational encoder embedding and CLIP visual embedding with the clothing image. In the reverse process of the diffusion model provided by the present application, the Transformer block has no cross-attention layer, and each Transformer block performs variational encoder embedding and CLIP visual embedding with the clothing image, thereby achieving clothing-centric visual adaptation of the denoising process and improving the consistency of the details of the clothes and the person's posture in the target image.

[0191] Based on the same inventive concept of the virtual clothing changing method, this embodiment provides a computer-readable storage medium, and a computer program is stored thereon. The computer program is executed by a processor to implement the virtual clothing changing method.

[0192] Specifically,

[0193] Get person images and clothing images.

[0194] Generate a target image based on the diffusion model. The target image contains a person image wearing a clothing image.

[0195] Among them, the diffusion model includes forward process and reverse process.

[0196] The forward process generates a noisy image of the target image.

[0197] In the reverse process, the Transformer block has no cross-attention layer, and each Transformer block uses the clothing image for variational encoder embedding and CLIP visual embedding.

[0198] Optionally, the inverse process of the diffusion model includes:

[0199] The clothing image is deformed to obtain a deformed image.

[0200] Extract an irrelevant image from the person image that has nothing to do with clothing.

[0201] The noisy image is concatenated with the deformed image and the irrelevant image in the channel dimension.

[0202] The stitched image is used as input to the inverse process.

[0203] Optionally, the loss function used by the denoising network during the inverse of the diffusion model is

[0204] in, is the mean square error loss, is the appearance loss, and λ is a hyperparameter used to balance the mean square error loss and appearance loss.

[0205] ∈ is noise, ∈ θ (γ,I c ,I w ,t) predicted noise, γ is the spliced ​​image, I c is a clothing image, I w is the deformed image, t is the time step, and θ is the prediction parameter.

[0206] is the spatial perception loss, It is the high frequency enhancement loss.

[0207] is a function that captures the difference in structure and texture, ⊙ is a multiplication operator, is the predicted target image, I is the target image, m w is the deformation mask, m w It is obtained by deforming the clothing image.

[0208] For The edge image extracted from h is the edge image extracted from I.

[0209] Optionally, the variational encoder embeds f vae =MLP vae (ε(I w )).

[0210] Among them, MLP vae (·) is the multilayer perceptron embedded in the variational encoder, ε(·) is the variational encoder, I w is a deformed image, the I e It is obtained by deforming the clothing image.

[0211] Optionally, CLIP visual embedding f clip =MLP clip (CLIP v (I c )).

[0212] Among them, MLP clip (·) is the multi-layer perceptron of CLIP visual embedding, CLIP v (·) is the CLIP visual encoder, I c For clothing images.

[0213] Optionally, the criss-cross attention mechanism in the reverse process is:

[0214]

[0215] Among them, Z attn is the output of the cross attention mechanism, Softmax(·) is the normalization function, Q is the query in the cross attention, K clip is the key of CLIP, is the transpose, d is the dimension of the key, V clip is the value of CLIP, K vae is the key of the variational encoder, V vae is the value of the variational encoder.

[0216] Q=f q W q , f q is the feature of the previous self-attention layer, W q is the query projection matrix in the cross attention.

[0217] f clip is the CLIP feature, The key projection matrix for CLIP.

[0218] The projection matrix for the CLIP value.

[0219] f vae is the variational encoder feature, is the key projection matrix of the variational encoder.

[0220] is the value projection matrix of the variational encoder.

[0221] Optionally, the criss-cross attention mechanism in the reverse process is:

[0222]

[0223] Among them, Softmax(·) is the normalization function, Q is the query in the cross attention, and K clip is the key of CLIP, is the transpose, d is the dimension of the key, V clip is the value of CLIP, K vae is the key of the variational encoder, V vae is the value of the variational encoder, ⊙ is the multiplication operator, M attn is the attention mask obtained based on the deformation mask, where the deformation mask is obtained by deforming the clothing image.

[0224] Q=f q Wq , f q is the feature of the previous self-attention layer, W q is the query projection matrix in the cross attention.

[0225] f clip is the CLIP feature, The key projection matrix for CLIP.

[0226] The projection matrix for the CLIP value.

[0227] f vae is the variational encoder feature, is the key projection matrix of the variational encoder.

[0228] is the value projection matrix of the variational encoder.

[0229] The computer-readable storage medium provided in this embodiment, the computer program thereon is executed by a processor to obtain a person image and a clothing image; a target image is generated based on a diffusion model; the person image in the target image wears a clothing image; wherein the diffusion model includes a forward process and a reverse process; the forward process generates a noisy image of the target image; in the reverse process, the Transformer block has no cross-attention layer, and each Transformer block performs variational encoder embedding and CLIP visual embedding with the clothing image. In the reverse process of the diffusion model provided in the present application, the Transformer block has no cross-attention layer, and each Transformer block performs variational encoder embedding and CLIP visual embedding with the clothing image, thereby achieving clothing-centric visual adaptation of the denoising process and improving the consistency of the details of the clothes and the person's posture in the target image.

[0230] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0231] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0232] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0233] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 Stream

[0234] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0235] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A virtual clothing changing method, characterized in that: The method comprises: Obtain person images and clothing images; Generate a target image based on a diffusion model; the person image in the target image wears the clothing image; Wherein, the diffusion model includes a forward process and a reverse process; The forward process generates a noisy image of the target image; In the reverse process, the Transformer block has no cross-attention layer but includes a visual adapter; wherein each visual adapter is embedded with a variational encoder and a CLIP visual embedding with the clothing image; and each visual adapter is implemented using a cross-attention mechanism; The inverse process of the diffusion model includes: Deforming the clothing image to obtain a deformed image; Extracting an irrelevant image irrelevant to clothing from the person image; Mapping the deformed image and the irrelevant image to a latent space, and then concatenating them with the noisy image in a channel dimension; Use the stitched image as input to the reverse process; Among them, in the reverse process of the diffusion model, the loss function used by the denoising network is in, is the mean square error loss, is the appearance loss, and λ is a hyperparameter used to balance the mean square error loss and appearance loss; ∈ is noise, ∈ θ (γ,I c ,I w ,t) predicted noise, γ is the spliced ​​image, I c is a clothing image, I w is the deformed image, t is the time step, and θ is the prediction parameter; is the spatial perception loss, Enhance loss for high frequencies; is a function that captures the difference in structure and texture, ⊙ is a multiplication operator, is the predicted target image, i is the target image, m w is the deformation mask, m w is obtained by deforming the clothing image; For The edge image extracted from h is the edge image extracted from I; Among them, the variational encoder embeds f vae =MLP vae (ε(I w )); Among them, MLP vae (·) is the multilayer perceptron embedded in the variational encoder, ε(·) is the variational encoder, I w is a deformed image, the I w is obtained by deforming the clothing image; Among them, CLIP visual embedding f clip =MLP clip (CLIP v (I c )); Among them, MLP clip (·) is the multi-layer perceptron of CLIP visual embedding, CLIP v (·) is the CLIP visual encoder, I c is the clothing image; Among them, the cross attention mechanism in the reverse process is: Among them, Z attn is the output of the cross attention mechanism, Softmax(·) is the normalization function, Q is the query in the cross attention, K clip is the key of CLIP, T is the transpose, d is the dimension of the key, V clip is the value of CLIP, K vae is the key of the variational encoder, V vae is the value of the variational encoder; Q=f q W q , f q is the feature of the previous self-attention layer, W q is the query projection matrix in cross attention; f clip is the CLIP feature, is the key projection matrix of CLIP; The projection matrix for the CLIP value; f vae is the variational encoder feature, is the key projection matrix of the variational encoder; is the value projection matrix of the variational encoder; Among them, the cross attention mechanism in the reverse process is: Among them, Softmax(·) is the normalization function, Q is the query in the cross attention, and K clip is the key of CLIP, T is the transpose, d is the dimension of the key, V clip is the value of CLIP, K vae is the key of the variational encoder, V vae is the value of the variational encoder, ⊙ is the multiplication operator, M attn is an attention mask obtained based on a deformation mask, wherein the deformation mask is obtained by deforming the clothing image; Q=f q W q , f q is the feature of the previous self-attention layer, W q is the query projection matrix in cross attention; f clip is the CLIP feature, is the key projection matrix of CLIP; The projection matrix for the CLIP value; f vae is the variational encoder feature, is the key projection matrix of the variational encoder; is the value projection matrix of the variational encoder.

2. A virtual clothing changing device, characterized in that: The device comprises: An acquisition module, used for acquiring character images and clothing images; A generating module, configured to generate a target image based on a diffusion model; the target image includes the image of a person wearing the image of clothing; Wherein, the diffusion model includes a forward process and a reverse process; The forward process generates a noisy image of the target image; In the reverse process, the Transformer block has no cross-attention layer but includes a visual adapter; wherein each visual adapter is embedded with a variational encoder and a CLIP visual embedding with the clothing image; and each visual adapter is implemented using a cross-attention mechanism; The inverse process of the diffusion model includes: Deforming the clothing image to obtain a deformed image; Extracting an irrelevant image irrelevant to clothing from the person image; Mapping the deformed image and the irrelevant image to a latent space, and then concatenating them with the noisy image in a channel dimension; Use the stitched image as input to the reverse process; Among them, in the reverse process of the diffusion model, the loss function used by the denoising network is in, is the mean square error loss, is the appearance loss, and λ is a hyperparameter used to balance the mean square error loss and appearance loss; ∈ is noise, ∈ θ (γ,I c ,I w ,t) predicted noise, γ is the spliced ​​image, I c is a clothing image, I w is the deformed image, t is the time step, and θ is the prediction parameter; is the spatial perception loss, Enhance loss for high frequencies; is a function that captures the difference in structure and texture, ⊙ is a multiplication operator, is the predicted target image, I is the target image, m w is the deformation mask, m w is obtained by deforming the clothing image; For The edge image extracted from h is the edge image extracted from I; Among them, the variational encoder embeds f vae =MLP vae (ε(I w )); Among them, MLP vae (·) is the multilayer perceptron embedded in the variational encoder, ε(·) is the variational encoder, I w is a deformed image, the I w is obtained by deforming the clothing image; Among them, CLIP visual embedding f clip =MLP clip (CLIP v (I c )); Among them, MLP clip (·) is the multi-layer perceptron of CLIP visual embedding, CLIP v (·) is the CLIP visual encoder, I c is the clothing image; Among them, the cross attention mechanism in the reverse process is: Among them, Z attn is the output of the cross attention mechanism, Softmax(·) is the normalization function, Q is the query in the cross attention, K clip is the key of CLIP, T is the transpose, d is the dimension of the key, V clip is the value of CLIP, K vae is the key of the variational encoder, V vae is the value of the variational encoder; Q=f q W q , f q is the feature of the previous self-attention layer, W q is the query projection matrix in cross attention; f clip is the CLIP feature, is the key projection matrix of CLIP; The projection matrix for the CLIP value; f vae is the variational encoder feature, is the key projection matrix of the variational encoder; is the value projection matrix of the variational encoder; Among them, the cross attention mechanism in the reverse process is: Among them, Softmax(·) is the normalization function, Q is the query in the cross attention, and K clip is the key of CLIP, T is the transpose, d is the dimension of the key, V clip is the value of CLIP, K vae is the key of the variational encoder, V vae is the value of the variational encoder, ⊙ is the multiplication operator, M attn is an attention mask obtained based on a deformation mask, wherein the deformation mask is obtained by deforming the clothing image; Q=f q W q , f q is the feature of the previous self-attention layer, W q is the query projection matrix in cross attention; f clip is the CLIP feature, is the key projection matrix of CLIP; The projection matrix for the CLIP value; f vae is the variational encoder feature, is the key projection matrix of the variational encoder; is the value projection matrix of the variational encoder.

3. An electronic device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to claim 1.

4. A computer-readable storage medium, characterized in that: A computer program is stored thereon; the computer program is executed by a processor to implement the method according to claim 1.

Images