An image style transfer method based on a diffusion model

Abstract

The application discloses an image style transfer method based on a diffusion model, which first uses a CLIP image encoder and an attention mechanism to obtain the text embedding of a style image, and performs style transfer on the text embedding to obtain a more accurate semantic representation of the style image. Then, the forward process of the diffusion model is performed on the content image, and the text embedding of the style image is added in the reverse process. The application proposes a hybrid denoising module in the reverse process, which has two U-Net modules, and the tasks of the two U-Net modules are to maintain the structural features of the content image and generate the artistic style of the style image respectively. The method aims to ensure the accuracy of the style image transfer, and combines the semantic and visual features of the content and style images to generate a style transfer image. The method can solve the problem that the generative adversarial network may cause blurring or artifacts, and overcome the difficult challenges such as gradient disappearance and mode collapse in the training process.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an image style transfer method based on a diffusion model. Background Technology

[0002] Style transfer is a key task in computer vision and image processing, aiming to fuse the content of one image with the style of another to create a completely new image. The challenge lies in simultaneously preserving the structure and details of the content image and the stylistic features of the fused style image. Generative Adversarial Networks (GANs) are widely used for style transfer; however, training GAN models is challenging, often accompanied by problems such as vanishing gradients and mode collapse. Furthermore, most GAN networks may generate blur or artifacts, affecting the quality of the style-transferred image. The paper "An Improved CycleGAN Image Style Transfer Algorithm [J]. Journal of Beijing Institute of Printing, 2023, 31(09):1-6" discloses an image style transfer algorithm using a GAN network. The GAN network is trained through adversarial interaction between the generator and discriminator. This method first constructs a generator based on U-Net, then creates a discriminator using the PatchGAN structure, and introduces self-attention mechanisms in both the generator and discriminator to further improve the attention to and reconstruction of key details. The method described in the literature is a style transfer model based on a GAN network, but the generated images contain blurred regions, limiting its practical application value. Furthermore, the constant competition between the generator and discriminator can lead to instability in the training process and rapid changes in gradients, potentially causing gradient vanishing problems. Therefore, further improvement and research are important topics in the field of image style transfer. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention provides an image style transfer method based on a diffusion model. First, a CLIP image encoder and attention mechanism are used to obtain the text embeddings of the style image, and style transfer is performed on the text embeddings to obtain a more accurate semantic representation of the style image. Then, the content image undergoes a forward process using the diffusion model, and the text embeddings of the style image are added during the reverse process. This invention proposes a hybrid denoising module in the reverse process, which includes two U-Net modules, tasked with maintaining the structural features of the content image and generating the artistic style of the style image, respectively. This method aims to ensure the accuracy of style image transfer while combining the semantic and visual features of both the content and style images to generate a style-transferred image. This invention's method can solve the problem of blurring or artifacts that may be caused by generative adversarial networks and overcomes the difficulties and challenges of gradient vanishing and mode collapse during training.

[0004] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0005] Step 1: Input style image y to obtain text embedding v0;

[0006] Step 1-1: Input a style image y, and use the CLIP image encoder to embed the style image into the text space to generate embedding features τ. θ (y);

[0007] Steps 1-2: Use two convolutional layers to extract shallow features, and then... θ (y) is converted to v′ and v respectively. i ;

[0008] Steps 1-3: Extract deep features of the text through an N-layer cross-attention module, with v′ and v′ at each layer. i Entering the cross-attention module, we obtain v i-1 During this process, v′ remains unchanged, as follows:

[0009] Q = W q v′,K=W k v i V = W v v i (1)

[0010]

[0011] In the formula W q W k W v Let d be the weight matrix of the mapping. k is the dimension of matrix K, which is used to scale the denominator of the dot product calculation to control the scale of the attention weights; v′ and v i After passing through N layers of cross-attention modules, the text embedding v0 is obtained;

[0012] Step 2: Perform style migration on the text embedding v0;

[0013] This is achieved through the following loss function:

[0014]

[0015] In the formula, x0 is the initial sample sampled from the distribution q(x0), w is the style vector adjusted through iterations during the optimization process, ∈ θ (x t ,t,MultiAtt(τ θ (y))) is a style adjustment function that indicates the text embedding to be adjusted;

[0016] Step 3: Input the content image x, and diffuse it using DDIM (Denoising Diffusion Implicit Models) to obtain random noise x. T ;

[0017] Input a content image x, and through the forward pass of DDIM, Gaussian noise is gradually added to the content image to obtain random noise x. T ;

[0018] DDIM is a diffusion model consisting of two processes: a forward process and a backward process. The forward process of DDIM is the noise-adding process, which is considered a Markov process, as follows:

[0019]

[0020]

[0021] Where, x 1:T Represents an image sequence containing all states from time step 1 to time step T, x t Let I represent the image at step t, and let α represent the identity matrix. t This represents the noise figure at step t. Represent a normal distribution function;

[0022] Step 4: Use a CDM-based hybrid denoising module to denoise the noise x. T Perform hybrid denoising to generate a style-transferred image z;

[0023] In the image generation process, the reverse process of the DDIM diffusion model is used; in the reverse process, a hybrid denoising module (CDM) is constructed.

[0024] The hybrid noise reduction module (CDM) consists of two U-Net modules, one of which takes random noise x as input. T Its task is to maintain the structural features of the content image; another U-Net module handles the random noise x in the content image. T Based on this, the text embedding information v0 of the style image is received as input;

[0025] The formula for the hybrid denoising function is as follows:

[0026]

[0027] In the formula, ∈ θ (x t Let p(x) be an unconditional image distribution, and λ(t) be the set of binding coefficients at different time steps t; c represents the content image, and I s Image representing style;

[0028] Using a hybrid denoising function for sampling is equivalent to sampling from a conditional distribution:

[0029]

[0030] Where p(x|I) c p(x|I) represents the image distribution given the content image. s ) represents the image distribution given a style image condition, and p(x) represents the unconditional image distribution;

[0031] Step 5: Compare the style-transferred image z with the two input images to construct a loss function and optimize the diffusion model;

[0032] When creating the loss function, two types of losses were created using the content image and the style image: content loss and style loss. and style loss

[0033] Content loss The goal is to match the similarity between the generated output image z and the input content image x, based on the features extracted from VGG-16 features; style loss. Its function is to match the mean and standard deviation of the VGG-16 features between the generated output image z and the input style image y; the formula for combining these two losses is as follows:

[0034]

[0035] In the formula, α and β are the weight parameters of the loss function. By adjusting their magnitudes, the relative importance of the two losses can be controlled, thereby controlling the effect of the generated image.

[0036] Preferably, the image encoder is a CLIP image encoder.

[0037] Preferably, w is typically initialized as a normal distribution N(0,1).

[0038] The beneficial effects of this invention are as follows:

[0039] To overcome the shortcomings of existing image style transfer algorithms based on generative adversarial networks (GANs), this invention proposes an image style transfer method based on a diffusion model. The proposed method aims to effectively generate a single image from a style image and a content image by utilizing a diffusion model, preserving both the structure and details of the original content image and the artistic style of the style image. This method addresses the blurring or artifacts that may result from GANs and overcomes challenges such as gradient vanishing and mode collapse during training. Attached Figure Description

[0040] Figure 1 This is a flowchart of the style transfer process in this invention.

[0041] Figure 2 Image y represents the style of an embodiment of the present invention.

[0042] Figure 3 Image x represents the content of an embodiment of the present invention.

[0043] Figure 4 The style transfer image z is predicted in an embodiment of the present invention. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0045] Taking the input style image y and content image x as an example, such as Figure 2 , Figure 3 The output style-transferred image z, such as Figure 4 This describes the specific implementation method. In this embodiment, the image style transfer method based on diffusion model training is as follows: Figure 1 As shown, it includes the following steps:

[0046] Step 1: Input the style image y to obtain the text embedding v0.

[0047] Given a style image y, the CLIP image encoder is first used to embed the style image into the text space, generating embedding features τ. θ (y). CLIP is a powerful image-text learning model that establishes deep semantic connections between text and images through contrastive learning. Then, two convolutional layers are used to extract shallow features, and τ... θ (y) is converted to v′ and v respectively. i Subsequently, deep features of the text are extracted through an N-layer cross-attention module, with each layer's v′ and v i Entering the cross-attention module, we obtain v i-1 During this process, v′ remains unchanged, as follows:

[0048] Q = W q v′,K=W k v i V = W v v i (1)

[0049]

[0050] In the formula W q W k W v Let d be the weight matrix of the mapping. kis the dimension of matrix K, used to scale the denominator of the dot product calculation to control the scale of the attention weights. v′ and v i After passing through N layers of cross-attention modules, the text embedding v0 is obtained.

[0051] Step 2: Before embedding into the diffusion model, perform style transfer on the text embedding v0.

[0052] The goal of style transfer is to adjust the style features of text embeddings while preserving the semantic information of the embeddings. This can be achieved using the following loss function:

[0053]

[0054] In the formula, x0 is the initial sample sampled from the distribution q(x0). w is the style vector adjusted through iterations during the optimization process, typically initialized to a normal distribution N(0, I). ∈ θ (x t ,t,MultiAtt(τ θ (y))) is the style adjustment function, representing the adjustment of the text embedding to better match the target style. By continuously optimizing the parameter θ, the loss function tries to minimize this distance, thus making the text embedding more consistent with the target style.

[0055] Step 3: Input the content image x, and diffuse it using DDIM to obtain random noise x. T .

[0056] The input image x is processed through the forward pass of DDIM to generate random noise. DDIM (Denoising Diffusion Implicit Models) is an iterative implicit diffusion model that generates noise by performing noise diffusion (forward pass) and denoising (backward pass) on the data. The diffusion model consists of two processes: the forward pass and the backward pass. The forward pass, also known as the diffusion process, is the noise addition process and can be considered a Markov process, as follows:

[0057]

[0058]

[0059] The state x transitions between different time steps using a Gaussian distribution. The model parameters α... t and α t-1 It can be used to adjust the uncertainty and correlation of state evolution. This generation process is accomplished by sampling from an estimated probability distribution at each step, which is adjusted at each step, ultimately yielding the noise x. T .

[0060] Step 4: Utilize a CDM-based hybrid denoising module to denoise the noise x. T Perform hybrid denoising to generate a style-transferred image z.

[0061] In the image generation process, this method uses the inverse process of the diffusion model. During the inverse process, the method constructs a hybrid denoising module (CDM). The CDM consists of two U-Net modules, one of which takes random noise x from the content image as input. T One module's task is to maintain the structural features of the content image; another module handles the random noise x in the content image. T Based on this, the text embedding information v0 of the style image is input to ensure that the generated image has style features. The formula for the hybrid denoising function is as follows:

[0062]

[0063] In the formula, ∈ θ (x t Let p(x) be an unconditional image distribution, and λ(t) be the set of binding coefficients at different time steps t. Sampling using this hybrid denoising function is equivalent to sampling from a conditional distribution.

[0064]

[0065] Where p(x|I) c p(x|I) represents the image distribution given the content image. s Let denot be the image distribution given a style image, and p(x) be the unconditional image distribution. The goal of this process is to generate images while preserving content and style features. The trade-off between content and style is controlled by adjusting the binding coefficient λ(t) to achieve the desired image generation effect.

[0066] Step 5: Compare the style-transferred image z with the two input images to construct a loss function and optimize the diffusion model.

[0067] When creating the loss function, two types of losses were created using the content image and the style image: content loss and style loss. and style loss Content loss The goal is to match the similarity between features extracted from VGG-16 features in the generated output image z and the input content image x. This method relatively loosely preserves semantics and spatial layout, rather than simply matching pixel values. Style loss The purpose of this method is to match the mean and standard deviation of the VGG-16 features between the generated output image z and the input style image y. This helps ensure that the generated image is stylistically similar to the input style image. The formula for combining these two losses is as follows:

[0068]

[0069] In the formula, α and β are weight parameters of the loss function, and their magnitudes can be adjusted to control the relative importance of the two losses, thereby controlling the quality of the generated image. This method allows for image transformation in the target style while preserving content features, providing greater control and higher quality generated results.

Claims

1. An image style transfer method based on a diffusion model, characterized in that, Includes the following steps; Step 1: Input style image y to obtain text embedding v0; Step 1-1: Input a style image y, and use the CLIP image encoder to embed the style image into the text space to generate embedding features τ. θ (y); Steps 1-2: Use two convolutional layers to extract shallow features, and then... θ (y) is converted to v′ and v respectively. i ; Steps 1-3: Extract deep features of the text through an N-layer cross-attention module, with v′ and v′ at each layer. i Entering the cross-attention module, we obtain v i-1 During this process, v′ remains unchanged, as follows: Q=W q v',K=W k v i ,V=W v v i (1) In the formula W q W k W v Let d be the weight matrix of the mapping. k is the dimension of matrix K, which is used to scale the denominator of the dot product calculation to control the scale of the attention weights; v′ and v i After passing through N layers of cross-attention modules, the text embedding v0 is obtained; Step 2: Perform style migration on the text embedding v0; This is achieved through the following loss function: In the formula, x0 is the initial sample sampled from the distribution q(x0), w is the style vector adjusted through iterations during the optimization process, ∈ θ (x t ,t,MultiAtt(τ θ (y))) is a style adjustment function that indicates the text embedding to be adjusted; Step 3: Input the content image x, and diffuse it using DDIM (Denoising Diffusion Implicit Models) to obtain random noise x. T ; Input a content image x, and through the forward pass of DDIM, Gaussian noise is gradually added to the content image to obtain random noise x. T ; DDIM is a diffusion model that includes two processes: a forward process and a reverse process. The forward pass of DDIM is a noise-adding process, which is considered a Markov process, as follows: Where, x 1:T Represents an image sequence containing all states from time step 1 to time step T, x t Let I represent the image at step t, and let α represent the identity matrix. t This represents the noise figure at step t. Represent a normal distribution function; Step 4: Use a CDM-based hybrid denoising module to denoise the noise x. T Perform hybrid denoising to generate a style-transferred image z; In the image generation process, the reverse process of the DDIM diffusion model is used; in the reverse process, a hybrid denoising module (CDM) is constructed. The hybrid noise reduction module (CDM) consists of two U-Net modules, one of which takes random noise x as input. T Its task is to maintain the structural features of the content image; another U-Net module handles the random noise x in the content image. T Based on this, the text embedding information v0 of the style image is received as input; The formula for the hybrid denoising function is as follows: In the formula, ∈ θ (x t Let λ(t) represent an unconditional image distribution p(x), and let λ(t) be the set of binding coefficients at different time steps t; c Represents a content image, I s Image representing style; Using a hybrid denoising function for sampling is equivalent to sampling from a conditional distribution: Where p(x|I) c p(x|I) represents the image distribution given the content image. s ) represents the image distribution given a style image condition, and p(x) represents the unconditional image distribution; Step 5: Compare the style-transferred image z with the two input images to construct a loss function and optimize the diffusion model; When creating the loss function, two types of losses were created using the content image and the style image: content loss and style loss. and style loss Content loss The goal is to match the similarity between the generated output image z and the input content image x, based on the features extracted from VGG-16 features; style loss. Its function is to match the mean and standard deviation of the VGG-16 features between the generated output image z and the input style image y; the formula for combining these two losses is as follows: In the formula, α and β are the weight parameters of the loss function. By adjusting their magnitudes, the relative importance of the two losses can be controlled, thereby controlling the effect of the generated image.

2. The image style transfer method based on a diffusion model according to claim 1, characterized in that, The image encoder is a CLIP image encoder.

3. The image style transfer method based on a diffusion model according to claim 1, characterized in that, The w is usually initialized as a normal distribution N(0,I).

Images