Decoupling type image style migration method based on diffusion model

Abstract

The invention provides a decoupling type image style migration method based on a diffusion model, and relates to the field of image processing, and the method comprises the steps: constructing an image style migration model based on content and style decoupling, the image style migration model at least comprises a content extraction module, a style extraction module, a projection network and a style migration module, and the model decouples contents and styles from a feature level and a network level at the same time; constructing a training set covering a plurality of data types; training the image style migration model through the training set and the loss function; based on the trained image style migration model, the content image and the style image are input, the stylized result image with the target content and the target style is generated, and the method has the advantages that the style migration effect is improved, and more interpretability is achieved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a decoupled image style transfer method based on a diffusion model. Background Art

[0002] Image style transfer is an important research direction in the field of computer vision. It aims to transfer the style features in one image to another image, thereby generating a new image with target content and target style. This technology has been widely used in the fields of artistic creation and entertainment. Especially in social networks, image style transfer is often used for image beautification, personalized artistic creation, and decoration of personal photos to enhance creativity and fun. In addition, image style transfer also plays an important role in the commercial field. Many well-known brands have applied it to advertising and product display to enhance visual appeal and strengthen marketing effects. As one of the core technologies in computer vision, image style transfer is not only of great significance in tasks such as image enhancement, data enhancement, and image translation, but also can provide more diverse training data by converting images of a specific style into other styles, thereby improving the generalization ability of the model. Overall, image style transfer technology provides users with a broad creative space and design possibilities, while opening up new application scenarios and development opportunities for various industries, and has a broad and far-reaching positive impact.

[0003] Before the widespread application of deep learning technology, traditional image style transfer methods mainly rely on manual modeling, and their core idea is to describe the texture of local features of the image through statistical models. Although these methods can achieve a certain degree of style transfer, they have significant limitations. For example, each model can only adapt to a single style, the generation process needs to be trained from scratch, and the style fusion effect is poor, which often leads to unnatural style transition in the generated image. With the rapid development of deep learning technology, style transfer methods have gradually turned to implementation based on convolutional neural networks (CNNs). This type of method defines the high-level features extracted by pre-trained convolutional neural networks as content, the correlation between features as style, and constructs corresponding content loss and style loss functions, which not only significantly improves the efficiency of image generation, but also significantly enhances the generalization ability of the model. However, since the learning process of convolutional neural networks is difficult to explain, this definition based on feature correlation fails to fully decouple image content and style, resulting in existing methods still having problems such as noise texture, color imbalance and semantic information loss in style transfer tasks, and it is difficult to achieve a perfect fusion of content and style. In recent years, diffusion models have made significant breakthroughs in the field of image generation and have demonstrated performance superior to traditional generation models. The diffusion model can generate higher quality images through a gradual denoising process and provide stronger capabilities in style control. Therefore, studying image style transfer methods based on the diffusion model is a topic of great significance. Summary of the invention

[0004] In order to solve the problems of noise texture, color imbalance and semantic information loss in the existing image style transfer methods, the present invention proposes a decoupled image style transfer method based on a diffusion model, comprising: constructing an image style transfer model based on content and style decoupling, wherein the image style transfer model at least includes a content extraction module, a style extraction module, a projection network and a style transfer module; constructing a training set covering multiple data types; training the image style transfer model through the training set and a loss function; based on the trained image style transfer model, inputting a content image and a style image to generate a stylized result image with target content and target style.

[0005] Furthermore, the content extraction module is used to extract content information from the image, and the specific steps are as follows: considering that color is an important component of style, the content extraction module first removes the color information of the image through a brightness transformation method to generate a grayscale image of the image; then, based on the principle that a pre-trained diffusion model can help eliminate specific domain features of the input image and align it with the pre-trained domain, the content extraction module uses the diffusion model pre-trained in the proxy domain to perform forward denoising and reverse denoising on the grayscale image, thereby removing style details such as brushstrokes and textures in the image, and finally extracting the content information of the image.

[0006] Furthermore, the style extraction module is used to extract the style representation from the style image. The specific process is as follows: given an input style image I s , extract the content of the style image through the content extraction module The original style image I s Input to the CLIP image encoder E to obtain a global representation f containing style and content s , the content of the style image Input to CLIP image encoder E to obtain the content representation of the style image Use the global representation f s Subtract content representation Get style representation Its expression is: Among them, given that the content of an image can usually be easily identified by its structure, semantics or shape, while the definition of style is relatively vague, style is considered to be a supplement to content, thereby achieving a complete decoupling of content and style. This decoupling method transforms the control of style into the control of content extraction. Compared with previous methods, it not only has higher controllability, but also significantly enhances the interpretability of the model.

[0007] Furthermore, the projection network is used to map the style representation into style features aligned with the text feature dimensions in the pre-trained diffusion model. The projection network consists of a linear layer and a normalization layer. The linear layer is used to align the dimensions, and the normalization layer is used to improve training stability and convergence speed.

[0008] Furthermore, the style transfer module is used to perform style transfer according to the content features of the content image and the style features of the style image. A new cross-attention layer is added to the module where the diffusion model captures the style information. The content features and the style features are respectively injected into the denoising process of the diffusion model by combining the original cross-attention layer and the newly added cross-attention layer. The content features of the content image are inserted through the original cross-attention layer of the pre-trained diffusion model, and the expression is:

[0009] Z c =Attention(Q,K c ,V c ),

[0010] Where Q = ZW q is the query weight W of the cross-attention layer composed of the query feature Z and the pre-trained diffusion model q The calculated query matrix, K c =c c W k The content feature c c and the key weights W of the cross-attention layer of the pre-trained diffusion model k The calculated bond matrix, V c =c c W v is determined by the content feature c c and the value weight W of the cross attention layer of the pre-trained diffusion model v The calculated value matrix;

[0011] The style features of the style image inserted through the new cross-attention layer are expressed as:

[0012] Z s =Attention(Q,K s ,V s ),

[0013] Where Q = ZW q is the query weight W of the cross-attention layer composed of the query feature Z and the pre-trained diffusion model q The calculated query matrix, K s =c s W k ′ is composed of style feature c s and the newly added trainable key weights W k ′The calculated bond matrix, Vs =c s W v ′ is composed of style feature c s and the newly added trainable value weight W v 'Calculated value matrix;

[0014] The output of the original cross-attention layer and the output of the new cross-attention layer are added to combine the content features and style features, and the expression is: Z new =Z c +Z s .

[0015] Furthermore, the construction of a training set covering multiple data types is used to improve the performance of the image style transfer model from the perspective of data diversity. Considering that diverse data can significantly enhance the generalization ability of the model and promote its application in various style transfer tasks, the present invention collects images of multiple style types and constructs two types of image pairs: one is an image pair copied from a single image, and the other is an image pair consisting of a content image and a style image with the same style but different content. By training on the first type of image pairs, the model can be guaranteed to have high-quality generation capabilities; and by training on the second type of image pairs, the style learning ability of the model can be effectively enhanced.

[0016] Furthermore, the training of the image style transfer model by the training set and the loss function includes: inputting the content image into the content extraction module, the content extraction module extracting the content representation of the content image; inputting the style image into the style extraction module, the style extraction module extracting the style representation of the style image; inputting the style representation into the projection network, the projection network mapping the style representation into the style feature aligned with the text feature dimension in the pre-trained diffusion model; inputting the content image into the VAE encoder to obtain the image representation of the content image in the latent space, and performing random noise processing on the image representation of the content image to obtain the initial potential noise of the subsequent diffusion process; inputting the content representation, the style feature and the initial potential noise into the diffusion model of the style transfer module, and the model predicts the noise; by using the same loss function as the original diffusion model, calculating the error loss between the predicted noise and the actual noise; according to the loss, optimizing the trainable parameters of the image style transfer model by back propagation; the loss function expression is:

[0017]

[0018] Where, ∈ is the random noise added when the image representation of the content image is randomly denoised, ∈ θ (z t ,t,c) is the neural network ∈ for denoising θAccording to the time step t, the noise input z at time step t t , the noise predicted by the conditional cue c including the content representation of the content image and the style features of the style image.

[0019] Furthermore, the trainable parameters of the image style transfer model include all parameters of the projection network and the key weights W of the new cross-attention layer added by the style transfer module. k ′ and value weight W v ′, the parameters of other modules of the model are set according to the pre-trained model and do not need to be updated.

[0020] Furthermore, the stylized result image with target content and target style is generated based on the trained image style transfer model, and the specific process is: inputting the content image into the content extraction module to extract the content representation of the content image; inputting the style image into the style extraction module to extract the style representation of the style image; inputting the style representation into the trained projection network to obtain the style features aligned with the text feature dimensions in the pre-trained diffusion model; inputting the content image into the VAE encoder to obtain the image representation of the content image in the latent space, and performing specific noise addition processing on the image representation of the content image to obtain the initial latent noise of the subsequent diffusion process; inputting the content representation, style features and initial latent noise into the trained style transfer module to obtain the style transfer features; decoding the style transfer features to obtain the style transfer image, the content of the style transfer image is consistent with the content image, and the style is consistent with the style image.

[0021] Compared with the existing methods, the decoupled image style transfer method based on the diffusion model provided by the present invention has at least the following beneficial effects:

[0022] 1. Aiming at the problems of noise texture, transition migration, color unevenness, loss of semantic information of original image, etc. in the existing style transfer methods, this paper proposes a new style transfer method based on the diffusion model. With the powerful generation ability of the diffusion model, the stylized result image has higher quality;

[0023] 2. A new style transfer framework that decouples image content and style is proposed. The problem of difficult-to-define image style is solved by explicitly extracting content information and implicitly learning its complementary style information. Since the style is strictly constrained as a supplement to the content, the content and style can be completely decoupled at the feature level. At the same time, two independent cross-attention layers are used to insert content features and style features respectively, and the style features are only injected into specific style blocks, so that the content and style can also be decoupled at the network level;

[0024] 3. The entire model requires fewer parameters to be trained. During the training phase, only the parameters of the projection network and the new cross-attention layer added in the style transfer module are trained. Other modules use the settings of the pre-trained model and do not need to be updated. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Attached Figure 1 It is a flowchart of a decoupled image style transfer method based on a diffusion model proposed in the present invention.

[0026] Attached Figure 2 It is a flow chart of a content extraction module of a decoupled image style transfer method based on a diffusion model proposed in the present invention.

[0027] Attached Figure 3 It is a structural schematic diagram of a style extraction module of a decoupled image style transfer method based on a diffusion model proposed in the present invention.

[0028] Attached Figure 4 It is a structural schematic diagram of a projection network of a decoupled image style transfer method based on a diffusion model proposed in the present invention.

[0029] Attached Figure 5 It is a schematic diagram of the overall network structure of a decoupled image style transfer method based on a diffusion model proposed in the present invention. DETAILED DESCRIPTION

[0030] A typical embodiment of a decoupled image style transfer method based on a diffusion model of the present invention is described in detail below in conjunction with the accompanying drawings, and the present invention is further specifically explained to help those skilled in the art understand the features and advantages of the present invention more clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0031] Attached Figure 1 is a flow chart of a decoupled image style transfer method based on a diffusion model proposed in the present invention. Figure 1 As shown, a decoupled image style transfer method based on a diffusion model may include the following process:

[0032] Step 1: construct an image style transfer model based on content and style decoupling, wherein the image style transfer model at least includes a content extraction module, a style extraction module, a projection network and a style transfer module.

[0033] Attached Figure 2 is a flow chart of a content extraction module of a decoupled image style transfer method based on a diffusion model proposed in the present invention, such as Figure 2 As shown in the figure, the content extraction module is used to extract content information from an image. The specific process is as follows: given an input image, its color is removed by a brightness transformation method to obtain a grayscale image of the image; a pre-trained diffusion model is used to perform forward denoising and reverse denoising on the grayscale image to remove style details such as brush strokes and textures of the grayscale image, and extract the content of the image.

[0034] Attached Figure 3 is a structural schematic diagram of a style extraction module of a decoupled image style transfer method based on a diffusion model proposed in the present invention, such as Figure 3 As shown in the figure, the style extraction module is used to extract the style representation from the style image. The specific process is as follows: given an input style image I s , extract the content of the style image through the content extraction module The original style image I s Input to the CLIP image encoder E to obtain a global representation f containing style and content s , the content of the style image Input to CLIP image encoder E to obtain the content representation of the style image Use the global representation f s Subtract content representation Get style representation Its expression is:

[0035] Attached Figure 4 is a schematic diagram of the structure of a projection network of a decoupled image style transfer method based on a diffusion model proposed in the present invention, such as Figure 4 As shown, the projection network is used to map the style representation into style features that are aligned with the text feature dimensions in the pre-trained diffusion model. The projection network consists of a linear layer and a normalization layer. The linear layer is used to align the dimensions of the style features and the text features, and the normalization layer is used to improve the training stability and convergence speed.

[0036] The style transfer module of the decoupled image style transfer method based on the diffusion model proposed in the present invention is used to perform style transfer according to the content features of the content image and the style features of the style image. A new cross-attention layer is added to the module where the diffusion model captures the style information. The content features and the style features are respectively injected into the denoising process of the diffusion model by combining the original cross-attention layer and the newly added cross-attention layer. The content features of the content image are inserted through the original cross-attention layer of the pre-trained diffusion model, and the expression is:

[0037] Z c =Attention(Q,K c ,V c ),

[0038] Where Q = ZW q is the query weight W of the cross-attention layer composed of the query feature Z and the pre-trained diffusion model q The calculated query matrix, K c =c c W k is determined by the content feature c c and the key weights W of the cross-attention layer of the pre-trained diffusion model k The calculated bond matrix, V c =c c W v is determined by the content feature c c and the value weight W of the cross attention layer of the pre-trained diffusion model v The calculated value matrix;

[0039] The style features of the style image inserted through the new cross-attention layer are expressed as:

[0040] Z s =Attention(Q,K s ,V s ),

[0041] Where Q = ZW q is the query weight W of the cross-attention layer composed of the query feature Z and the pre-trained diffusion model q The calculated query matrix, K s =c s W k ′ is composed of style feature c s and the newly added trainable key weights W k ′The calculated bond matrix, V s =c s W v ′ is composed of style feature c s and the newly added trainable value weight W v 'Calculated value matrix;

[0042] The output of the original cross-attention layer and the output of the new cross-attention layer are added to combine the content features and style features, and the expression is: Z new =Z c +Z s .

[0043] Step 2: Build a training set covering multiple data types.

[0044] Construct a training set covering multiple data types to improve the performance of the image style transfer model from the perspective of data diversity. Collect images of multiple style types and construct two image pairs (content image, style image). One is an image pair copied from an image, and the other is an image pair consisting of a content image and a style image with the same style as the content image but inconsistent content.

[0045] Step three: Train the image style transfer model using the training set and loss function.

[0046] The image style transfer model is trained by using a training set and a loss function, including: inputting a content image into a content extraction module, the content extraction module extracting a content representation of the content image; inputting a style image into a style extraction module, the style extraction module extracting a style representation of the style image; inputting the style representation into a projection network, the projection network mapping the style representation into a style feature aligned with a text feature dimension in a pre-trained diffusion model; inputting the content image into a VAE encoder to obtain an image representation of the content image in a latent space, performing random noise processing on the image representation of the content image to obtain an initial latent noise for a subsequent diffusion process; inputting the content representation, the style feature and the initial latent noise into a diffusion model of the style transfer module, the model predicting the noise; calculating the loss by using the same loss function as the original diffusion model; and optimizing the trainable parameters of the image style transfer model by back propagation according to the loss.

[0047] Furthermore, the trainable parameters of the image style transfer model include all parameters of the projection network and the key weights W of the new cross-attention layer added in the style transfer module. k ′ and value weight W v ′, the parameters of other modules of the model are set according to the pre-trained model and do not need to be updated.

[0048] Step 4: Based on the trained image style transfer model, input the content image and style image to generate a stylized result image with target content and target style.

[0049] Attached Figure 5 : is a schematic diagram of the overall network structure of a decoupled image style transfer method based on a diffusion model proposed in the present invention. Figure 5As shown, a stylized result image with target content and target style is generated based on a trained image style transfer model, and the specific process is: inputting the content image into a content extraction module to extract the content representation of the content image; inputting the style image into a style extraction module to extract the style representation of the style image; inputting the style representation into a trained projection network to obtain style features aligned with the text feature dimensions in a pre-trained diffusion model; inputting the content image into a VAE encoder to obtain an image representation of the content image in a latent space, and performing specific noise addition processing on the image representation of the content image to obtain the initial latent noise for a subsequent diffusion process; inputting the content representation, style features, and initial latent noise into a trained style transfer module to obtain post-style transfer features; decoding the post-style transfer features to obtain a post-style transfer image, the content of the post-style transfer image being consistent with the content image, and the style being consistent with the style image.

[0050] In the example of the present invention, the images used in network training are from the WikiArt dataset, including abstract expressionism, impressionism, primitivism, neo-impressionism, neoclassicism and post-impressionism, etc. The image format is JPG format, and all images are resized to 512*512 by using the resize function of the Image module in the PIL library of Python. The brightness transformation method used to remove the image color in the content extraction module is ITUR 601-2, and the specific formula is Y=0.299*R+0.587*G+0.114*B, where R represents the red component in the image, G represents the green component in the image, and B represents the blue component in the image. The diffusion model used in the content extraction module is the ADM diffusion model pre-trained on ImageNet. Since the images in the ImageNet dataset are all photos, the input images of different domains can be converted into potential domains through the forward process through the ADM diffusion model, and then reversely converted into the photo domain through the reverse process, eliminating the style features of the image, leaving only the content of the image. The forward denoising process and the backward denoising process of the content extraction module both use the DDIM sampling technique, where the maximum time step number T is set to 600 and the number of samples S is set to 40. The CLIP image encoder in the style extraction module and the overall network framework are both encoders in OpenCLIP ViT-H / 14. The diffusion model used in the style transfer process is SDv1.5. The style transfer module only introduces a new cross-attention layer in the part of the diffusion model that captures style information, corresponding to the third layer of the downsampling stage, the middle block, and the second layer of the upsampling stage in the SDv1.5 model. During the training process, the AdamW optimizer is used to update the corresponding weight gradients, with a fixed learning rate of 0.0001 and a weight decay of 0.01.

[0051] It is necessary to point out here that the above embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Ordinary technical personnel in this field may make some non-essential improvements and adjustments to these embodiments without departing from the principle of the present invention, which still fall within the scope of protection of the present invention.

Claims

1. A decoupled image style transfer method based on a diffusion model, characterized in that: include: Constructing an image style transfer model based on content and style decoupling, wherein the image style transfer model at least includes a content extraction module, a style extraction module, a projection network and a style transfer module; Build a training set covering a variety of data types; Train the image style transfer model using training sets and loss functions; Based on the trained image style transfer model, the content image and style image are input to generate a stylized result image with target content and target style.

2. According to the decoupled image style transfer method based on diffusion model in claim 1, it is characterized in that: The content extraction module is used to extract content information from the image. The specific process is as follows: a. Given an input image, remove its color through brightness transformation method to obtain the grayscale image of the image; b. Use the pre-trained diffusion model to perform forward denoising and reverse denoising on the grayscale image to remove style details such as brush strokes and textures of the grayscale image and extract the content of the image.

3. According to the decoupled image style transfer method based on diffusion model in claim 1, it is characterized in that: The style extraction module is used to extract style representation from the style image. The specific process is as follows: a. Given an input style image, extract the content of the style image through the content extraction module; b. Input the original style image into the CLIP image encoder to obtain a global representation containing style and content. Input the content of the style image into the CLIP image encoder to obtain the content representation of the style image. Subtract the content representation from the global representation to obtain the style representation, which is expressed as: fs=E(Is), fsc=E(Isc), fss=fs-fsc, Among them, Is represents the style image, Isc represents the content of the style image, E represents the CLIP image encoder, fs represents the global representation of the style image, fsc represents the content representation of the style image, and fss represents the style representation of the style image.

4. The decoupled image style transfer method based on a diffusion model according to claim 1, characterized in that: The projection network is used to project the style representation into a style feature aligned with the text feature dimension in the pre-trained diffusion model. The projection network consists of a linear layer and a normalization layer. The linear layer is used to align the dimension, and the normalization layer is used to improve the training stability and convergence speed.

5. The decoupled image style transfer method based on diffusion model according to claim 1, characterized in that: The style transfer module is used to perform style transfer according to the content features of the content image and the style features of the style image. A new cross-attention layer is added to the module where the diffusion model captures the style information. The content features and style features are respectively injected into the denoising process of the diffusion model by combining the original cross-attention layer and the newly added cross-attention layer. The content features of the content image are inserted through the original cross-attention layer of the pre-trained diffusion model, and the expression is: Zc=Attention(Q,Kc,Vc), Wherein, Q=ZWq is the query matrix calculated by the query feature Z and the query weight Wq of the cross attention layer of the pre-trained diffusion model, Kc=ccWk is the key matrix calculated by the content feature cc and the key weight Wk of the cross attention layer of the pre-trained diffusion model, and Vc=ccWv is the value matrix calculated by the content feature cc and the value weight Wv of the cross attention layer of the pre-trained diffusion model; The style features of the style image inserted through the new cross-attention layer are expressed as: Zs=Attention(Q,Ks,Vs), Where Q = ZWq is the query matrix calculated by the query feature Z and the query weight Wq of the cross-attention layer of the pre-trained diffusion model, Ks = csWk′ is the key matrix calculated by the style feature cs and the newly added trainable key weight Wk′, and Vs = csWv′ is the value matrix calculated by the style feature cs and the newly added trainable value weight Wv′; The output of the original cross-attention layer and the output of the new cross-attention layer are added to combine the content features and style features, and the expression is: Znew = Zc + Zs.

6. The decoupled image style transfer method based on diffusion model according to claim 1, characterized in that: The method of constructing a training set covering multiple data types is used to improve the performance of the image style transfer model in terms of data diversity. Images of multiple style types are collected, and two (content image, style image) image pairs are constructed, one of which is an image pair copied from an image, and the other is an image pair consisting of a content image and a style image that is consistent with the content image in style but inconsistent in content.

7. The decoupled image style transfer method based on diffusion model according to claim 1, characterized in that: The training of the image style transfer model by using a training set and a loss function includes: a. inputting the content image into a content extraction module, and the content extraction module extracts the content representation of the content image; b. Input the style image into the style extraction module, and the style extraction module extracts the style representation of the style image; c. Input the style representation into the projection network, which maps the style representation into style features aligned with the text feature dimensions in the pre-trained diffusion model; d. Input the content image into the VAE encoder to obtain the image representation of the content image in the latent space, and perform random noise processing on the image representation of the content image to obtain the initial latent noise for the subsequent diffusion process; e. Input the content representation, style features, and initial potential noise into the diffusion model of the style transfer module, and the model predicts the noise; f. Calculate the loss by using the same loss function as the original diffusion model; g. Based on the loss, optimize the trainable parameters of the image style transfer model through back propagation.

8. The decoupled image style transfer method based on diffusion model according to claim 7, characterized in that: The trainable parameters of the image style transfer model include all parameters of the projection network and the key weights Wk′ and value weights Wv′ of the new cross-attention layer added in the style transfer module. The parameters of other modules of the model are set by the pre-trained model and do not need to be updated.

9. The decoupled image style transfer method based on diffusion model according to claim 1, characterized in that: The specific process of generating a stylized result image with target content and target style based on the trained image style transfer model is as follows: a. Inputting the content image into the content extraction module to extract the content representation of the content image; b. Input the style image into the style extraction module to extract the style representation of the style image; c. Input the style representation into the trained projection network to obtain style features that are aligned with the text feature dimensions in the pre-trained diffusion model; d. Input the content image into the VAE encoder to obtain the image representation of the content image in the latent space, and perform specific noise processing on the image representation of the content image to obtain the initial latent noise for the subsequent diffusion process; e. Input the content representation, style features, and initial potential noise into the trained style transfer module to obtain the style transfer features; f. Decoding the style-transferred features to obtain a style-transferred image, wherein the content of the style-transferred image is consistent with the content image, and the style is consistent with the style image.

Images