Image stylization method based on diffusion model
Through the diffusion model and self-attention mechanism, combined with noise processing and text conditions, the problem of insufficient generation capabilities and large amount of calculation in text-driven style transfer is solved, and high-quality style transfer and structural fidelity are achieved.
Patent Information
- Application Number
- CN202411023647.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-22
AI Technical Summary
In the prior art, the method of text-driven style transfer task is based on the GAN model, resulting in limited modeling and generation capabilities between text and style, making it difficult to retain the global content of the original image, and the training process requires a large amount of computing and large-scale text-guided images. Training tuples.
Using a diffusion model-based image stylization method, the noise diffusion model DDIM and text conditional stable diffusion model LDM are used, combined with self-attention mechanism and negative prompts, spatial features are manipulated to generate high-quality style images, reducing the computational volume and training needs. .
It improves image quality and fidelity of the original image structure, reduces the calculation amount and training needs, and achieves high-quality style transfer effects without the need for a large amount of data training.
Smart Images

Figure CN119006310A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of image processing, and in particular to an image stylization method based on a diffusion model. Background Art
[0002] Image style transfer is attractive in practice because it allows amateurs to transform real-world photos into renderings that mimic the style of artworks without requiring any professional skills. Previous style transfer methods are based on extracted style textures, which transfer the semantic texture of the style image to the content image and generate vivid artworks.
[0003] The development of GAN (Gan Adversarial Neural Network) has brought new ideas to image style transfer technology. GAN can generate realistic images. Combining GAN (Gan Adversarial Neural Network) technology with image style transfer can achieve higher quality and more realistic style transfer effects. Pioneering methods based on deep learning have proposed various GAN (Gan Adversarial Neural Network) based frameworks to encourage the output image to conform to the distribution of the target domain. However, these methods require example image datasets from the source and target domains, and usually need to be trained from scratch for each style transfer task (e.g., horse to zebra, day to night, summer to winter). At the same time, GAN-based methods rely on example image dataset discriminators and cannot generate satisfactory results when the example image dataset is inappropriate.
[0004] As diffusion model-based methods have evolved, they have demonstrated stunning results on vision tasks, especially in generating artworks. Today, diffusion model-based generators can synthesize images of higher quality than GAN-based ones.
[0005] CN117689532A discloses an image style transfer method based on a diffusion model, which uses a CLIP image encoder and an attention mechanism to obtain text embedding of a style image, and performs style transfer on the text embedding to obtain a more accurate semantic representation of the style image. Then, the forward process of the diffusion model is performed on the content image, and the text embedding of the style image is added in the reverse process.
[0006] (1) Ruiz proposed a new approach to text-to-image diffusion modeling. Given only a few images of a subject as input, they fine-tuned a pre-trained text-to-image model so that it learns to bind a unique identifier to that specific subject. Once the subject is embedded in the output domain of the model, the unique identifier can be used to synthesize novel realistic images of the subject in different scenes.
[0007] In the prior art, there are often the following problems:
[0008] Disadvantage 1: Current methods for text-driven style transfer tasks are mainly based on GAN models, which leads to limited modeling and generation capabilities between text and style.
[0009] Disadvantage 2: The style transfer task needs to preserve the original image structure, but the current diffusion work cannot preserve the global content of the original image well.
[0010] Disadvantage 3: Existing technologies require a lot of computation and large-scale text-guided image training tuples when training a text-to-image model. Summary of the invention
[0011] The purpose of the present invention is to overcome the above-mentioned deficiencies of the prior art and to provide an image stylization method based on a diffusion model.
[0012] An image stylization method based on a diffusion model comprises the following steps:
[0013] S1, input original image;
[0014] S2, the original image is processed into an upper step and a lower step, and both the upper step and the lower step are restored to the initial Gaussian noise image in the forward process;
[0015] S3, the initial Gaussian noise image in the previous step is gradually denoised using the noise diffusion implicit model DDIM in the reverse process to obtain the original denoised image, and then the spatial features are extracted;
[0016] S4, the initial Gaussian noise image in the next step is gradually denoised using the text conditional stable diffusion model LDM in the reverse process, and the spatial features obtained in step S3 and the input guide text are added to output the style image.
[0017] The diffusion implicit model DDIM is a probabilistic generative model where images are generated by T ~N(0,I) is generated by gradually removing noise. In the forward process, Gaussian noise is gradually added to the clean image x0. The formula is as follows:
[0018]
[0019] Among them z~N(0,I) and {α t} is the noise table, x T is the initial Gaussian noise image, Z is the noise, and t is the time step.
[0020] The image is also processed in the reverse process. The purpose of the reverse process is to gradually T Denoising is performed, where each step results in a clearer image, which is done by predicting the neural network ∈ θ(x t ,t) to achieve, after the training is completed, each step of the reverse process includes converting ∈ θ Applied to the current x t , and remove the Gaussian noise disturbance to obtain a clearer and cleaner x t-1 .
[0021] The text conditional stable diffusion model LDM is based on the U-Net architecture, which is conditioned by the guide prompt P. The U-Net architecture includes residual blocks, self-attention blocks and cross-attention blocks. The residual blocks are used to detect image features from the previous layer l-1. Perform convolution to generate intermediate features Given by the following formula:
[0022]
[0023]
[0024] in, For query value, For key values, is a single value, is the output of the self-attention block.
[0025] At each time step t in the reverse process of style image generation, the spatial features are extracted from the denoising step The spatial features obtained Injected into the generation of style transfer pictures, expressed as:
[0026]
[0027] Where X t * Indicates the current image, P indicates the guide text, use To indicate the use of injection features The modified denoising step, if not injected,
[0028] The self-attention module linearly projects the spatial features into the query value and the key value and then calculates the affinity between the spatial features. By the following formula:
[0029]
[0030] in Represents the spatial features of the 4th layer, and controls The maximum injection layer improves the fidelity of the original input image structure.
[0031] Use negative cues and describe the prompts that guide the image P n , at each sampling step, the final noise prediction becomes
[0032] in Given by the following formula:
[0033]
[0034] Where φ represents the output of the current U-Net architecture, α∈[0,1] is used to balance neutral cues and negative cues, and w is the scale without classifier guidance.
[0035] The present invention has the following beneficial technical effects:
[0036] The image style transfer task is realized based on the diffusion model, and the generated image quality is higher than that of the traditional GAN. Compared with the commonly used diffusion model, this invention improves the fidelity of the original image structure by manipulating the spatial features and self-attention within the model, and alleviates the problem of unnecessary appearance leakage of the original image. n In addition, the stylization of the style image is improved by using the parameter α∈[0,1] to balance the neutral cues and negative cues.
[0037] Conventional image style transfer training requires large-scale data images for training, while the present invention manipulates the spatial features and self-attention within the model, which does not require a large amount of calculation and large-scale text-guided image training tuples, greatly reducing the amount of calculation and training. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 It is a schematic diagram of the process of the present invention;
[0039] Figure 2 Schematic diagram of the model of the U-Net architecture of the present invention. DETAILED DESCRIPTION
[0040] In order to further understand the features, technical means, specific objectives and functions of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0041] An image stylization method based on a diffusion model comprises the following steps:
[0042] S1, input original images, which contain diverse images of various human images in different styles, including real images and text-generated images.
[0043] S2, the original image is processed in the upper step and the lower step, and both the upper step and the lower step are restored to the initial Gaussian noise image in the forward process. The upper step and the lower step here are in a parallel relationship, that is, the original input image is processed in two steps at the same time.
[0044] S3, the initial Gaussian noise image in the previous step is gradually denoised using the noise diffusion implicit model DDIM in the reverse process to obtain the original denoised image, and then the spatial features are extracted.
[0045] S4, the initial Gaussian noise image in the next step is gradually denoised using the text conditional stable diffusion model LDM in the reverse process, and the spatial features obtained in step S3 and the input guide text are added to output the style image.
[0046] The diffusion implicit model DDIM is a probabilistic generative model where images are generated by T ~N(0,I) is generated by gradually removing noise. In the forward process, Gaussian noise is gradually added to the clean image x0. The formula is as follows:
[0047]
[0048] Among them z~N(0,I) and {α t} is the noise table, x T is the initial Gaussian noise image, Z is the noise, and t is the time step.
[0049] The image is also processed in the reverse process. The purpose of the reverse process is to gradually T Denoising is performed, where each step results in a clearer image, which is done by predicting the neural network ∈ θ (x t ,t) to achieve, after the training is completed, each step of the reverse process includes ∈ θ Applied to the current x t , and remove the Gaussian noise disturbance to obtain a clearer and cleaner x t-1 .
[0050] The text conditional stable diffusion model LDM is based on the U-Net architecture, where the diffusion process is applied to the latent space of a pre-trained image autoencoder. Conditioned on the guidance prompt P, the U-Net architecture includes residual blocks, self-attention blocks, and cross-attention blocks. The residual blocks pay attention to the image features from the previous layer l-1. Perform convolution to generate intermediate features Given by the following formula:
[0051]
[0052]
[0053] in, For query value, For key values, is a single value, is the output of the self-attention block. In order to better understand how the semantic space information is θ In the internal encoding, a PCA (principal component analysis) is performed, which can infer that in ∈ θ Specifically, we use a diverse set of images containing various human-like images of different styles, including real images and text-generated images. For each image, we extract features from each layer of the decoder at each time step t. PCA is then applied to all images. The result is that the coarsest and shallowest layers are dominated by foreground-background separation, depicting only rough spots in the locations of foreground objects. Subsequently, it can be observed that the intermediate features (i.e., layer 4) encode local semantic information shared between objects from different domains.
[0054] At each time step t in the reverse process of style image generation, the spatial features {f} are extracted from the denoising step: The spatial features obtained Injected into the generation of style transfer pictures, expressed as:
[0055]
[0056] in Indicates the current image, P indicates the guide text, use To indicate the use of injection features The modified denoising step, if not injected, Experiments show that injecting features only in the 4th layer is not enough to preserve the structure of the guidance image. If features are injected in deeper layers, the guidance image structure is better preserved, but the appearance information of the original image leaks into the generated image. In order to achieve a better balance between preserving the original image structure and style transfer, consider not modifying the deep spatial features, but using the self-attention layer discussed below.
[0057] The self-attention module linearly projects the spatial features into the query value and key value and then calculates the affinity between the spatial features. The self-attention module linearly projects the spatial features into the query value and the key value and then calculates the affinity between the spatial features. By the following formula:
[0058]
[0059] in Represents the spatial features of the 4th layer, and controls The maximum injection layer improves the fidelity of the original input image structure and alleviates the problem of unnecessary appearance leakage of the original image.
[0060] Use negative cues and describe the prompts that guide the image P n , at each sampling step, the final noise prediction becomes in Given by the following formula:
[0061]
[0062] Where φ represents the output of the current U-Net architecture, α∈[0,1] is used to balance neutral cues and negative cues, and w is the scale without classifier guidance.
[0063] Through the above processing, the following features are achieved:
[0064] 1. Use the residual block in the U-Net architecture to extract the image features from the previous layer (l-1) Perform convolution to generate intermediate features Using a diverse set of images containing various human-like images of different styles, including real images and text-generated images, for each image, extract features from each layer of the decoder at each time step t Then apply PCA to all images.
[0065] 2. Improve the fidelity of the original image structure and optimize the problem of unnecessary appearance leakage of the original image. Extract spatial features from the denoising step These features are then injected into the generation of style transfer images, and a self-attention mechanism is added to achieve a better balance between preserving the guided image structure and style transfer.
[0066] 3. Optimize the noise prediction link. In order to improve the quality of style transfer, negative hints and hints describing the original image P are used. n In addition, the neutral and negative cues are balanced by using the parameter α∈[0,1].
[0067] 4. Perform high-quality text-guided image style transfer without any training or fine-tuning, and without style images.
[0068] It should be noted that the above description is not a limitation of the present invention. Without departing from the creative concept of the present invention, any obvious replacement is within the protection scope of the present invention.
Claims
1. An image stylization method based on diffusion model, It is characterized in that The following steps are involved: S1, input original image; S2, the original image is processed into an upper step and a lower step, and both the upper step and the lower step are restored to the initial Gaussian noise image in the forward process; S3, the initial Gaussian noise image in the previous step is gradually denoised using the noise diffusion implicit model DDIM in the reverse process to obtain the original denoised image, and then the spatial features are extracted; S4, the initial Gaussian noise image in the next step is gradually denoised using the text conditional stable diffusion model LDM in the reverse process, and the spatial features obtained in step S3 and the input guide text are added to output the style image.
2. The image stylization method based on the diffusion model according to claim 1, It is characterized in that The diffusion implicit model DDIM is a probabilistic generative model where images are generated by T ~N(0,I) is generated by gradually removing noise. In the forward process, Gaussian noise is gradually added to the clean image x 0 The formula is expressed as follows: Among them z~N(0,I) and {α t } is the noise table, x T is the initial Gaussian noise image, Z is the noise, and t is the time step.
3. The image stylization method based on the diffusion model according to claim 2, It is characterized in that The image is also processed in the reverse process. The purpose of the reverse process is to gradually T Denoising is performed, where each step results in a clearer image, which is done by predicting the neural network ∈ θ (x t ,t) to achieve, after the training is completed, each step of the reverse process includes converting ∈ θ Applied to the current x t , and remove the Gaussian noise disturbance to obtain a clearer and cleaner x t-1 .
4. The image stylization method based on the diffusion model according to claim 3, It is characterized in that The text conditional stable diffusion model LDM is based on the U-Net architecture, which is conditioned by the guide prompt P. The U-Net architecture includes residual blocks, self-attention blocks and cross-attention blocks. The residual blocks are used to detect image features from the previous layer l-1. Convolution is performed to produce the intermediate feature f t l , given by the following formula: Among them, q t l is the query value, k t l is the key value, t l is a single value, A t l is the output of the self-attention block.
5. The image stylization method based on the diffusion model according to claim 4, It is characterized in that At each time step t in the reverse process of style image generation, the spatial features are extracted from the denoising step The spatial features obtained Injected into the generation of style transfer pictures, expressed as: Where X t * Indicates the current image, P indicates the guide text, use To indicate the use of injection features The modified denoising step, if not injected, 6. The image stylization method based on diffusion model according to claim 5, It is characterized in that The self-attention module linearly projects the spatial features into the query value and the key value and then calculates the affinity A between the spatial features. t l , through the following formula: in Represents the spatial features of the 4th layer, and by controlling A t l The maximum injection layer improves the fidelity of the original input image structure.
7. The image stylization method based on diffusion model according to claim 6, It is characterized in that Use negative cues and describe the prompts that guide the image P n , at each sampling step, the final noise prediction becomes in Given by the following formula: in represents the output of the current U-Net architecture, α∈[0,1] is used to balance neutral and negative prompts, and w is the scale without classifier guidance.
Citation Information
Cited By
Multi-condition guide text image generation method based on decoupling and multi-domain guide strategy
CN120047565A
Secondary reconstruction image generation method based on conditional diffusion model
CN120599062A
Style migration method and system based on text inversion and self-attention injection
CN121033214A
