Text-guided image style migration method based on pre-training diffusion model

Abstract

The invention discloses a text-guided image style migration method based on a pre-trained diffusion model. The method comprises the following steps: acquiring a diffusion model, acquiring an input content image, and inputting the input content image into an auto-encoder for encoding to obtain a hidden space representation; performing DDIM inversion to obtain noise representation # imgabs0 # of the hidden space and a content characteristic spectrum corresponding to each step of the DDIM inversion; inputting the input style text into a CLIP text encoder, and obtaining a feature word vector of the input style text in a feature space; the method comprises the following steps: initializing random noise, inputting the random noise into a backbone network, performing DDIM reverse diffusion to obtain a style characteristic spectrum, and recording output of the backbone network in each reverse diffusion process; and inputting the noise representation of the hidden space as initial noise into the backbone network, performing DDIM reverse diffusion to obtain the hidden space representation of the stylized image, and inputting the hidden space representation into a decoder to obtain the stylized image. The method is more flexible in an actual application scene, and the limitation of a traditional method can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a text-guided image style transfer method based on a training diffusion model. Background Art

[0002] As a key research area in computer vision, style transfer has long garnered widespread attention from both academia and industry. Its core goal is to combine the content of one image with the artistic style of another, generating a new image that retains the semantic information of the content image while also incorporating the visual features of the style image. Early style transfer algorithms primarily relied on manually designed feature extractors, such as Gaussian filters and Laplacian pyramids. These methods extract low-level image features (such as edges and textures) through hand-crafted filters and then fuse the content and style images in pixel space. However, these methods have significant limitations. First, manually designed feature extractors can only capture low-level image features and are unable to effectively represent the semantic information and complex style of an image. Second, the effectiveness of style transfer is highly dependent on parameter adjustments. Changes in style require redesigning or optimizing computational parameters, resulting in poor flexibility and adaptability, making them difficult to meet the needs of practical applications. With the rapid development of deep learning technology, particularly the remarkable performance of convolutional neural networks (CNNs) in image feature extraction, style transfer research has achieved significant breakthroughs. In 2016, Gatys et al. first introduced CNNs to the style transfer task and proposed a feature space optimization method based on deep neural networks. This method uses pre-trained CNN models (such as the VGG network) to extract high-level semantic features from images and then matches and optimizes the features of the content image and style image in the feature space, thereby achieving high-quality style transfer. The innovation of this method lies in that it no longer relies on manually designed feature extractors, but instead automatically learns the semantic information and style features of images through deep neural networks, significantly improving the effectiveness and flexibility of style transfer. The work of Gatys et al. laid an important foundation for style transfer research. Many subsequent studies have revolved around this framework and proposed various improved methods, such as multi-style transfer and arbitrary style transfer.

[0003] The evolution of generative models has revolved around two core challenges: improving generation quality and optimizing computational efficiency. In the Generative Adversarial Network (GAN) framework, the interactive optimization of the generator and discriminator based on an adversarial training mechanism can synthesize visually realistic samples. However, its implicit probabilistic modeling characteristics can easily lead to mode collapse, significantly reducing the diversity of generated samples. In contrast, variational autoencoders (VAEs) achieve explicit probabilistic modeling by maximizing the evidence lower bound, but the deviation between their approximate posterior distribution and the true distribution often leads to the loss of high-frequency details in the generated results. The breakthrough of diffusion models is reflected in the reconstruction of the data generation process as an inverse solution of a differentiable physical system. Specifically, the forward diffusion process implements multi-step noise addition through a Markov chain, gradually degenerating the data distribution into an isotropic Gaussian distribution. The reverse process uses a parameterized denoising network to learn conditional probability transfer and iteratively reconstruct the target data distribution. This progressive generation paradigm offers two advantages: First, it overcomes the inherent stability flaws of the GAN framework by avoiding the vanishing / exploding gradient problem in adversarial training. Second, its explicitly defined probability density function ensures the distribution coverage of generated samples, thereby theoretically ensuring diversity. Research has shown that the diffusion model improves image texture detail fidelity metrics (such as FID and IS) compared to traditional methods.

[0004] However, the existing related technologies still have the following problems:

[0005] 1. Traditional image style transfer methods mostly rely on independently trained models. In particular, for diffusion model methods, the training process consumes a lot of computing resources, and when introducing new styles or filters, the model usually needs to be incrementally trained or retrained, which to some extent limits its wide application.

[0006] 2. These image-guided style transfer methods all require the input of a style image to guide the style change of the content image, and this limitation manifests itself as insufficient flexibility in practical applications.

[0007] 3. Existing diffusion model-based style transfer methods are insufficient in controlling the precision of style transfer.

[0008] 4. Since the diffusion model is a model that generates images based on random initial noise, the generated images are relatively random. Therefore, the current style transfer method based on the diffusion model has the problem of unstable stylization effect. Summary of the Invention

[0009] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of the present invention is to provide a text-guided image style transfer method based on a pre-trained diffusion model to solve the problems of high training cost, poor flexibility in style selection, difficult to control the style transfer process, and insufficient stability of style transfer in existing style transfer technologies.

[0010] The purpose of the present invention is achieved through the following technical solutions:

[0011] A text-guided image style transfer method based on a pre-trained diffusion model, comprising:

[0012] Obtaining a diffusion model, wherein the diffusion model includes three parts: an autoencoder, a decoder, and a backbone network;

[0013] Get the input content image and process it as follows:

[0014] Get the representation of the input content image in RGB space, input it into the autoencoder for encoding, and get the latent space representation

[0015] The latent space is represented As the starting point of DDIM inversion, it is input into the backbone network to obtain the noise representation of the input content image in the latent space. And the content feature map corresponding to each step of DDIM inversion

[0016] Get the input style text and process it as follows:

[0017] Input the input style text into the CLIP text encoder to obtain its feature word vector in the feature space;

[0018] Initialize random noise, input the random noise into the backbone network, and obtain the style feature map through DDIM reverse diffusion as well as At the same time, the output of the backbone network is recorded during each step of reverse diffusion.

[0019] The noise in the latent space is represented as As the initial noise input backbone network, DDIM reverse diffusion is performed to generate a stylized image and obtain the latent space representation of the stylized image

[0020] The latent space representation of the stylized image Input the decoder to obtain a stylized image.

[0021] Furthermore, the autoencoder samples a multi-stage downsampling structure, including three downsampling modules, a residual module and an intermediate processing module.

[0022] Furthermore, the decoder includes three upsampling modules, a residual module and an intermediate processing module.

[0023] Furthermore, the backbone network adopts a symmetrically designed Encoder-Decoder structure, wherein the Encoder structure includes 6 Spatial Transformers, the Decoder includes 6 Spatial Transformers, and each Spatial Transformer includes a residual block, a self-attention module, and a cross-attention module.

[0024] Furthermore, a downsampling factor is set, and the autoencoder maps the image I from the pixel space to the latent space, where h = H / f, w = W / f, and c is the number of channels of the autoencoder.

[0025] Furthermore, the latent space of the content image is represented As the starting point of DDIM inversion, it is input into the backbone network to obtain the noise representation of the input content image in the latent space. Specifically:

[0026] The latent space representation of the content image As the starting point of DDIM inversion, it is input into the backbone network. After T=50 steps of DDIM inversion, its noise representation in the latent space is obtained.

[0027] At the same time, record the content feature map of the self-attention module in each Spatial Transformer in the Decoder of the backbone network in each step. Where t represents the tth step of DDIM inversion.

[0028] Furthermore, random noise is initialized and input into the backbone network. After reverse diffusion through DDIM, the style feature map is obtained. Specifically:

[0029] Initialize random noise and input the random noise into the backbone network. After T=50 steps of DDIM reverse diffusion, input the text feature word vector in each reverse diffusion process and record the style feature map of the self-attention module in each Spatial Transformer in the Decoder of the backbone network in each reverse diffusion process. as well as At the same time, it is also necessary to record the output of the backbone network during each step of reverse diffusion.

[0030] Furthermore, the noise in the latent space is represented as As the initial noise input backbone network, DDIM reverse diffusion is performed to generate a stylized image and obtain its latent space representation Specifically:

[0031] At step \(t = 0\), parameter initialization is performed. Set the number of DDIM reverse diffusion steps \(T=50\), set the limit step length \(L = 10\) for step - limit adaptive instance normalization, and initialize the latent space noise representation.

[0032] Input the latent space noise representation of the current step into the backbone network to start the \(t\) - th step of DDIM reverse diffusion.

[0033] Diffusion feature injection: In DDIM reverse diffusion, inject the content feature map and style feature map into the self - attention module of each Spatial Transformer in the Decoder of the backbone network. and

[0034] If the current step \(t\lt L\), perform step - limit adaptive instance normalization; otherwise, skip this step and proceed to the next operation.

[0035] If the current step \(t\lt T\), set the current step \(t=t + 1\), and then jump back to start the \(t\) - th step of DDIM reverse diffusion.

[0036] The DDIM reverse diffusion is completed, and the latent space representation of the stylized image is obtained.

[0037] Furthermore, the diffusion feature injection is specifically as follows:

[0038] Replace the and in the self - attention module of each Spatial Transformer in the Decoder of the backbone network in two steps:

[0039] Content feature injection: Replace with That is, let

[0040] Style feature injection: Replace and [[ID= $47$]] with and That is, let

[0041]

[0042] Furthermore, the step - limit adaptive instance normalization is specifically as follows:

[0042] First, extract the mean and variance of the output result of the backbone network in the current DDIM reverse diffusion step.

[0043] At the same time, random noise is extracted and input into the backbone network, and the output result of the corresponding number of steps recorded by DDIM reverse diffusion is obtained. The mean and standard deviation

[0044] Then perform adaptive instance normalization:

[0045]

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] This paper proposes a style transfer method based on a pre-trained diffusion model. By utilizing a pre-trained diffusion model (such as Stable Diffusion) trained on a large-scale dataset, the entire style transfer framework does not require any additional training, thus achieving zero training overhead.

[0048] The present invention introduces the CLIP text encoder to integrate text information into the image generation process of the diffusion model, thereby realizing text-guided style transfer.

[0049] This method injects content and style features into the DDIM reverse diffusion process of the diffusion model through diffusion feature injection to achieve refined style transfer control.

[0050] This method enhances the stability of style transfer and the robustness of the model through step-limit adaptive instance normalization. Compared with traditional image guidance methods, this method is more flexible in practical application scenarios, can effectively address the limitations of traditional methods, and has wider applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a workflow diagram of the present invention;

[0052] FIG2(a) and FIG2(b) are schematic diagrams of the content and style feature map extraction of the present invention, respectively. DETAILED DESCRIPTION

[0053] The present invention will be further described in detail below with reference to the examples, but the embodiments of the present invention are not limited thereto.

[0054] Example

[0055] like Figure 1 As shown, the present invention provides a text-guided image style transfer method based on a pre-trained diffusion model, which specifically includes:

[0056] S1 obtains a diffusion model, which includes three parts: an autoencoder, a decoder, and a backbone network;

[0057] The diffusion model used in this embodiment is the open source Stable Diffusion v1.4, and its training process is as follows:

[0058] Based on large-scale image-text pair data (such as LAION-5B), by encoding the image into a latent space and performing diffusion modeling in this space, we learn how to guide the model to restore a clear image while gradually adding noise; the denoising process is completed by the backbone network, and the semantic conditions from the CLIP text encoder are combined at each step to make the generated image conform to the data distribution and fit the meaning of the input text; the entire training goal is to minimize the error between the predicted noise and the actual noise, thereby achieving efficient and controllable text-driven image generation capabilities.

[0059] The autoencoder and decoder are inverse processes of each other. The autoencoder adopts a multi-stage downsampling structure, including three downsampling modules, a residual module and an intermediate processing module. The decoder includes three upsampling modules, a residual module and an intermediate processing module.

[0060] The backbone network adopts a symmetrically designed Encoder-Decoder structure, where the Encoder structure includes 6 Spatial Transformers, the Decoder includes 6 Spatial Transformers, and each Spatial Transformer includes a residual block, a self-attention module, and a cross-attention module.

[0061] In this embodiment, the following diffusion models are all pre-trained diffusion models.

[0062] S2 obtains the content image that needs style transfer and performs the following processing, as shown in Figure 2(a):

[0063] S2.1 Input the content image C that needs to be style-transferred and obtain its representation I∈R in RGB space H×W×3 , where H and W are the width and height of the image in pixels.

[0064] S2.2 Input the representation in RGB space into the autoencoder for encoding to obtain the latent space representation

[0065] Specifically, this step uses an open-source pre-trained autoencoder E and sets a downsampling factor f. The autoencoder maps the image I from pixel space to latent space, where h = H / f and w = W / f. c is the number of channels in the autoencoder.

[0066] S2.3 will As the starting point of DDIM inversion (denoising diffusion implicit model inversion) input into StableDiffusion, after the number of steps T = 50 DDIM inversion steps, the noise representation in the latent space is obtained At the same time, record the feature maps of the self-attention module in each Spatial Transformer in the Decoder of the backbone network in each step. Where t represents the tth step of DDIM inversion. The process is shown in Figure 2(a), where the mathematical form of the tth step of DDIM inversion is:

[0067]

[0068] Among them, α t is the noise coefficient, ∈ θ Represents the noise prediction network. Through this process, the latent space representation of the content image at different time steps t∈{1,2...,T} can be obtained And the feature map in the self-attention module in the corresponding intermediate process

[0069] Further explanation, from the above formula, we can know that in the first step, t = 0, then according to the above formula, we can get get After S=50 steps, we actually get That is

[0070] S3 inputs the text S used to guide the style transfer of the content image and performs the following processing, as shown in Figure 2(b):

[0071] S3.1 Input the input style text into the CLIP text encoder to obtain its token in the feature space, namely the feature word vector; the CLIP text encoder can map the input text into the feature word vector.

[0072] S3.2 Initialize random noise and input the random noise into the Stable Diffusion backbone network. After T=50 steps of DDIM reverse diffusion, input the text token in each reverse diffusion step. Record the feature maps of the self-attention module in each Spatial Transformer in the Decoder of the backbone network during each reverse diffusion step. as well as At the same time, it is also necessary to record the output of Stable Diffusion in each step of reverse diffusion. Among them, for the t-th step DDIM reverse diffusion, its mathematical form is expressed as:

[0073]

[0074] Among them, α t is the noise coefficient, ∈ θ Represents the noise prediction network. Through this process, the latent space representation of the style text at different time steps t∈{1,2...,T} can be obtained And the feature map in the self-attention module in the corresponding intermediate process as well as

[0075] The DDIM reverse diffusion process is from get The process, that is, after 50 steps, can get

[0076] Noise representation of S4 in latent space As the initial noise, it is input into Stable Diffusion and subjected to DDIM reverse diffusion for T = 50 steps to generate a stylized image. The specific steps include the following:

[0077] S4.1 Step t = 0, perform parameter initialization. Set the DDIM reverse diffusion step number T = 50, set the limit step size L = 10 for step-limit adaptive instance normalization. Initialize the latent space noise representation

[0078] S4.2 represents the latent space noise of the current step Input into Stable Diffusion and start the t-th step of DDIM reverse diffusion;

[0079] S4.3 Diffusion Feature Injection: In the DDIM reverse diffusion, the feature maps of the content feature map and the style feature map are injected into the self-attention module of each SpatialTransformer in the Decoder of the backbone network. as well as

[0080] Specifically, the feature map of the self-attention module in Stable Diffusion is replaced. This process is carried out in two steps:

[0081] Content feature injection, Replace with That is

[0082] Style features injected and Replace with and That is

[0083] S4.4 If the current step number t < L, perform step - limit adaptive instance normalization; otherwise, skip this step and proceed to the next operation. Step - limit adaptive instance normalization: First, extract the mean and variance of the model output result in the current DDIM reverse diffusion step. At the same time, extract the mean and variance of the model output result corresponding to the step number recorded in step seven. Then perform adaptive instance normalization: The mean and variance extracted in this step are channel - level mean and variance.

[0084]

[0085]

[0086] S4.5 If the current step number t < T, set the current step number t = t + 1, and then jump to S4.2

[0087] S4.6 The DDIM reverse diffusion is completed, obtaining the latent space representation of the stylized image

[0088] S5: Input the latent space representation of the stylized image into the decoder D to obtain the stylized image I CS ∈R[[ID=​​​​​​​​​

Claims

1. A text-guided image style transfer method based on a pre-trained diffusion model, characterized in that: Including: Obtain a diffusion model, which includes three parts, namely an autoencoder, a decoder, and a backbone network; Obtain an input content image and perform the following processing: Get the representation of the input content image in RGB space, input it into the autoencoder for encoding, and get the latent space representation The latent space is represented As the starting point of DDIM inversion, it is input into the backbone network to obtain the noise representation of the input content image in the latent space. And the content feature map corresponding to each step of DDIM inversion Obtain an input style text and perform the following processing: Input the input style text into the CLIP text encoder to obtain its feature word vector in the feature space; Initialize random noise, input the random noise into the backbone network, and obtain the style feature map through DDIM reverse diffusion as well as At the same time, the output of the backbone network is recorded during each step of reverse diffusion. The noise in the latent space is represented as As the initial noise input backbone network, DDIM reverse diffusion is performed to generate a stylized image and obtain the latent space representation of the stylized image The latent space representation of the stylized image Input the decoder to obtain a stylized image.

2. The text-guided image style transfer method according to claim 1, characterized in that The autoencoder samples a multi-level downsampling structure, including three downsampling modules, a residual module, and an intermediate processing module.

3. The text-guided image style transfer method according to claim 1, characterized in that The decoder includes three upsampling modules, a residual module, and an intermediate processing module.

4. The text-guided image style transfer method according to any one of claims 1 to 3, characterized in that: The backbone network adopts a symmetrically designed Encoder-Decoder structure. Among them, the Encoder structure includes 6 SpatialTransformers, and the Decoder includes 6 SpatialTransformers. Each Spatial Transformer includes a residual block, a self-attention module, and a cross-attention module.

5. The text-guided image style transfer method according to claim 1, characterized in that Set a downsampling factor. The autoencoder maps the image I from the pixel space to the latent space, where h = H / f, w = W / f, and c is the number of channels of the autoencoder.

6. The text-guided image style transfer method according to claim 1, characterized in that The latent space representation of the content image As the starting point of DDIM inversion, it is input into the backbone network to obtain the noise representation of the input content image in the latent space. Specifically: The latent space representation of the content image As the starting point of DDIM inversion, it is input into the backbone network. After T=50 steps of DDIM inversion, its noise representation in the latent space is obtained. At the same time, record the content feature map of the self-attention module in each SpatialTransformer in the Decoder of the backbone network in each step. Where t represents the tth step of DDIM inversion.

7. The text-guided image style transfer method according to claim 1, characterized in that: Initialize random noise, input the random noise into the backbone network, and obtain the style feature map through DDIM reverse diffusion. Specifically: Initialize random noise and input the random noise into the backbone network. After T=50 steps of DDIM reverse diffusion, input the text feature word vector in each reverse diffusion process and record the style feature map of the self-attention module in each Spatial Transformer in the Decoder of the backbone network in each reverse diffusion process. as well as At the same time, it is also necessary to record the output of the backbone network during each step of reverse diffusion.

8. The text-guided image style transfer method according to claim 1, characterized in that: The noise in the latent space is represented as As the initial noise input backbone network, DDIM reverse diffusion is performed to generate a stylized image and obtain its latent space representation Specifically: At step t = 0, perform parameter initialization, set the DDIM reverse diffusion step number T = 50, set the limit step size L = 10 for step-limit adaptive instance normalization, and initialize the latent space noise representation The latent space noise of the current step is represented as Input into the backbone network and start the t-th step DDIM reverse diffusion; Diffusion feature injection: In DDIM reverse diffusion, the content feature map and style feature map are injected into the self-attention module of each SpatialTransformer in the Decoder of the backbone network as well as If the current step t < L, then perform step-limited adaptive instance normalization, otherwise skip this step and proceed to the next operation; If the current step t < T, then set the current step t = t + 1, and then jump back to the start of the t-step DDIM reverse diffusion; DDIM reverse diffusion is completed to obtain the latent space representation of the stylized image 9. The text-guided image style transfer method according to claim 8, characterized in that: The diffusion feature injection is specifically: The self-attention module of each Spatial Transformer in the Decoder of the backbone network and The replacement is done in two steps: Content feature injection, Replace with That is Style features injected and Replace with and That is 10. The text-guided image style transfer method according to claim 8, characterized in that: The step-limited adaptive instance normalization is specifically: First, extract the output of the backbone network in the current DDIM reverse diffusion step. The mean and variance At the same time, random noise is extracted and input into the backbone network, and the output result of the corresponding number of steps recorded by DDIM reverse diffusion is obtained. The mean and standard deviation Then perform adaptive instance normalization.

Images