A decoupled image style transfer method based on a diffusion model

Abstract

This invention proposes a decoupled image style transfer method based on a diffusion model, relating to the field of image processing. The method includes: constructing an image style transfer model based on content and style decoupling, wherein the image style transfer model includes at least a content extraction module, a style extraction module, a projection network, and a style transfer module, and the model decouples content and style simultaneously at the feature level and the network level; constructing a training set covering multiple data types; training the image style transfer model using the training set and a loss function; and, based on the trained image style transfer model, inputting a content image and a style image to generate a stylized result image with the target content and target style, which has the advantages of improving style transfer performance and enhancing interpretability.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a decoupled image style transfer method based on a diffusion model. Background Technology

[0002] Image style transfer is a significant research area in computer vision, aiming to transfer stylistic features from one image to another, thereby generating a new image with the target content and style. This technology has been widely applied in artistic creation and entertainment, especially on social networks, where it is frequently used for image enhancement, personalized artistic creation, and personal photo embellishment to enhance creativity and appeal. Furthermore, image style transfer plays a crucial role in the commercial sector, with many well-known brands utilizing it in advertising and product displays to improve visual appeal and strengthen marketing effectiveness. As a core technology in computer vision, image style transfer is not only significant in tasks such as image enhancement, data augmentation, and image translation, but it can also provide more diverse training data by transforming images of a specific style into others, thereby improving the generalization ability of models. Overall, image style transfer technology offers users a vast creative space and design possibilities, while simultaneously opening up new application scenarios and development opportunities for various industries, generating a broad and profound positive impact.

[0003] Before the widespread application of deep learning technology, traditional image style transfer methods mainly relied on manual modeling, the core idea of ​​which was to describe the texture of local features of an image through statistical models. Although these methods could achieve a certain degree of style transfer, they had significant limitations. For example, each model could only adapt to a single style, the generation process needed to be trained from scratch, and the style fusion effect was poor, often resulting in unnatural style transitions in the generated images. With the rapid development of deep learning technology, style transfer methods gradually shifted to implementations based on convolutional neural networks (CNNs). These methods define the high-level features extracted by pre-trained CNNs as content, the correlation between features as style, and construct corresponding content loss and style loss functions, which not only significantly improves the efficiency of image generation but also significantly enhances the generalization ability of the model. However, because the learning process of CNNs is difficult to interpret, this definition based on feature correlation fails to completely decouple image content and style, resulting in existing methods still having problems such as noisy textures, color imbalance, and loss of semantic information in style transfer tasks, making it difficult to achieve a perfect fusion of content and style. In recent years, diffusion models have made significant breakthroughs in the field of image generation, demonstrating performance superior to traditional generative models. Diffusion models, through a progressive denoising process, can generate higher-quality images and offer stronger style control capabilities. Therefore, researching image style transfer methods based on diffusion models is a topic of great significance. Summary of the Invention

[0004] To address the problems of noise texture, color imbalance, and semantic information loss in existing image style transfer methods, this invention proposes a decoupled image style transfer method based on a diffusion model. The method includes: constructing an image style transfer model based on content and style decoupling, wherein the image style transfer model includes at least a content extraction module, a style extraction module, a projection network, and a style transfer module; constructing a training set covering multiple data types; training the image style transfer model using the training set and a loss function; and, based on the trained image style transfer model, inputting a content image and a style image to generate a stylized result image with the target content and target style.

[0005] Furthermore, the content extraction module is used to extract content information from the image. The specific steps are as follows: Considering that color is an important component of style, the content extraction module first removes the color information of the image through a brightness transformation method to generate a grayscale image; then, based on the principle that a pre-trained diffusion model can help eliminate specific domain features of the input image and align it with the pre-trained domain, the content extraction module uses the diffusion model pre-trained in the proxy domain to perform forward noise addition and reverse noise reduction processing on the grayscale image, thereby removing stylistic details such as brushstrokes and textures in the image, and finally extracting the content information of the image.

[0006] Furthermore, the style extraction module is used to extract style representations from the style image, specifically as follows: given an input style image I... s Extract the content of the style image through the content extraction module. The original style image I s The input is fed into the CLIP image encoder E to obtain a global representation f containing style and content. s The content of the style image The input to CLIP image encoder E yields the content representation of the style image. f is represented globally. s Subtract content representation Obtaining style representation Its expression is: Given that image content can usually be easily identified through its structure, semantics, or shape, while the definition of style is relatively vague, style is considered as a complement to content, thus achieving complete decoupling between content and style. This decoupling transforms control over style into control over content extraction, offering not only greater controllability compared to previous methods but also significantly enhancing the interpretability of the model.

[0007] Furthermore, the projection network is used to map style representations to style features aligned with the text feature dimensions in the pre-trained diffusion model. The projection network consists of a linear layer and a normalization layer. The linear layer is used to align dimensions, and the normalization layer is used to improve training stability and convergence speed.

[0008] Furthermore, the style transfer module is used to perform style transfer based on the content features of the content image and the style features of the style image. This is achieved by adding a new cross-attention layer at the module that captures style information in the diffusion model. The content features and style features are then injected into the denoising process of the diffusion model by combining the original cross-attention layer and the newly added cross-attention layer. The expression for inserting the content features of the content image into the original cross-attention layer of the pre-trained diffusion model is as follows:

[0009] Z c =Attention(Q,K) c V c ),

[0010] Where Q = ZW q It consists of query features Z and query weights W from the cross-attention layer of the pre-trained diffusion model. q The calculated query matrix, K c =c c W k It is composed of content feature c c The key weights W of the cross-attention layer of the pre-trained diffusion model k The calculated key matrix, V c =c c W v It is composed of content feature c c The value weight W of the cross-attention layer of the pre-trained diffusion model v The calculated value matrix;

[0011] The style features of the style image inserted through the new cross-attention layer are expressed as follows:

[0012] Z s =Attention(Q,K) s V s ),

[0013] Where Q = ZW q It consists of query features Z and query weights W from the cross-attention layer of the pre-trained diffusion model. q The calculated query matrix, K s =c s W k ′ is composed of style feature c s And the newly added trainable key weights W k The calculated key matrix, Vs =c s W v ′ is composed of style feature c s And the newly added trainable value weights W v The calculated value matrix;

[0014] The outputs of the original cross-attention layer and the new cross-attention layer are added together to combine content and style features. The expression is: Z new =Z c +Z s .

[0015] Furthermore, the construction of a training set encompassing multiple data types is used to improve the performance of the image style transfer model through data diversity. Considering that diverse data can significantly enhance the model's generalization ability and promote its application in various style transfer tasks, this invention collects images of multiple style types and constructs two types of image pairs: one is an image pair obtained by copying a single image, and the other is an image pair consisting of a content image and a style image with the same style but different content. Training on the first type of image pair ensures the model possesses high-quality generation capabilities; while training on the second type of image pair effectively enhances the model's style learning ability.

[0016] Further, the training of the image style transfer model using a training set and a loss function includes: inputting the content image into a content extraction module, which extracts the content representation of the content image; inputting the style image into a style extraction module, which extracts the style representation of the style image; inputting the style representation into a projection network, which maps the style representation to style features aligned with the text feature dimensions in the pre-trained diffusion model; inputting the content image into a VAE encoder to obtain the image representation of the content image in the latent space, and randomly adding noise to the image representation of the content image to obtain the initial latent noise for the subsequent diffusion process; inputting the content representation, style features, and initial latent noise into the diffusion model of the style transfer module, where the model predicts the noise; calculating the error loss between the predicted noise and the actual noise using the same loss function as the original diffusion model; and optimizing the trainable parameters of the image style transfer model through backpropagation based on the loss. The expression for the loss function is:

[0017]

[0018] Where ∈ represents the random noise added during the random noise addition process to the image representation of the content image. θ (z t (t,c) is a neural network used for denoising. θBased on time step t, the noise input z at time step t t This includes content representation of the content image and conditional cues for style features of the style image, as well as noise prediction.

[0019] Furthermore, the trainable parameters of the image style transfer model include all parameters of the projection network and the key weights W of the new cross-attention layer added by the style transfer module. k Sum of weights W v The parameters of other modules of the model are the same as those of the pre-trained model and do not need to be updated.

[0020] Furthermore, the process of generating a stylized result image with target content and target style based on the trained image style transfer model is as follows: The content image is input into a content extraction module to extract its content representation; the style image is input into a style extraction module to extract its style representation; the style representation is input into a trained projection network to obtain style features aligned with the text feature dimensions in the pre-trained diffusion model; the content image is input into a VAE encoder to obtain its image representation in the latent space; specific noise addition processing is applied to the image representation of the content image to obtain initial latent noise for the subsequent diffusion process; the content representation, style features, and initial latent noise are input into a trained style transfer module to obtain style-transferred features; the style-transferred features are decoded to obtain a style-transferred image, wherein the content of the style-transferred image is consistent with the content image, and the style is consistent with the style image.

[0021] Compared to existing methods, the decoupled image style transfer method based on a diffusion model provided by this invention has at least the following advantages:

[0022] 1. To address the problems of noise texture, over-transfer, uneven color, and loss of semantic information of the original image in existing style transfer methods, this invention proposes a new style transfer method based on the diffusion model. With the powerful generation capability of the diffusion model, the stylized result image has higher quality.

[0023] 2. A novel style transfer framework that decouples image content and style is proposed. It solves the problem of the difficulty in defining image style by explicitly extracting content information and implicitly learning its complementary style information. Since style is strictly constrained as a supplement to content, content and style can be completely decoupled at the feature level. At the same time, two independent cross-attention layers are used to insert content features and style features respectively, and style features are injected only into specific style blocks, so that content and style can also be decoupled at the network level.

[0024] 3. The entire model requires fewer parameters to be trained. During the training phase, only the parameters of the projection network and the new cross-attention layer added in the style transfer module are trained. Other modules use the settings of the pre-trained model and do not need to be updated. Attached Figure Description

[0025] Appendix Figure 1 This is a flowchart illustrating a decoupled image style transfer method based on a diffusion model proposed in this invention.

[0026] Appendix Figure 2 This is a flowchart illustrating the content extraction module of a decoupled image style transfer method based on a diffusion model proposed in this invention.

[0027] Appendix Figure 3 This is a schematic diagram of the style extraction module of a decoupled image style transfer method based on a diffusion model proposed in this invention.

[0028] Appendix Figure 4 This is a schematic diagram of the projection network structure of a decoupled image style transfer method based on a diffusion model proposed in this invention.

[0029] Appendix Figure 5 This is a schematic diagram of the overall network structure of a decoupled image style transfer method based on a diffusion model proposed in this invention. Detailed Implementation

[0030] The following detailed description, with reference to the accompanying drawings, describes a typical embodiment of a decoupled image style transfer method based on a diffusion model according to the present invention, further illustrating the invention to help those skilled in the art to more clearly and completely understand the features and advantages of the invention. Obviously, the described embodiments are merely some embodiments of the present invention, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0031] Appendix Figure 1 This is a flowchart illustrating a decoupled image style transfer method based on a diffusion model proposed in this invention, as shown below. Figure 1 As shown, a decoupled image style transfer method based on a diffusion model may include the following steps:

[0032] Step 1: Construct an image style transfer model based on content and style decoupling. The image style transfer model includes at least a content extraction module, a style extraction module, a projection network, and a style transfer module.

[0033] Appendix Figure 2 This is a flowchart illustrating the content extraction module of a decoupled image style transfer method based on a diffusion model proposed in this invention. Figure 2 As shown, the content extraction module is used to extract content information from the image. The specific process is as follows: given the input image, its color is removed by the brightness transformation method to obtain the grayscale image; the grayscale image is then subjected to forward noise addition and reverse noise reduction processing using a pre-trained diffusion model to remove stylistic details such as brushstrokes and textures, thereby extracting the content of the image.

[0034] Appendix Figure 3 This is a schematic diagram of the style extraction module in a decoupled image style transfer method based on a diffusion model proposed in this invention. Figure 3 As shown, the style extraction module is used to extract style representations from style images. The specific process is as follows: Given an input style image I... s Extract the content of the style image through the content extraction module. The original style image I s The input is fed into the CLIP image encoder E to obtain a global representation f containing style and content. s The content of the style image The input to CLIP image encoder E yields the content representation of the style image. f is represented globally. s Subtract content representation Obtaining style representation Its expression is:

[0035] Appendix Figure 4 This is a schematic diagram of the projection network structure of a decoupled image style transfer method based on a diffusion model proposed in this invention, as shown below. Figure 4 As shown, the projection network is used to map style representations to style features aligned with the dimensions of text features in the pre-trained diffusion model. The projection network consists of a linear layer and a normalization layer. The linear layer is used to align the dimensions of style features and text features, and the normalization layer is used to improve training stability and convergence speed.

[0036] The style transfer module of the decoupled image style transfer method based on a diffusion model proposed in this invention performs style transfer based on the content features of the content image and the style features of the style image. A new cross-attention layer is added to the module that captures style information in the diffusion model. The content features and style features are then injected into the denoising process of the diffusion model by combining the original cross-attention layer and the newly added cross-attention layer. The expression for inserting the content features of the content image through the original cross-attention layer of the pre-trained diffusion model is as follows:

[0037] Z c =Attention(Q,K) c V c ),

[0038] Where Q = ZW q It consists of query features Z and query weights W from the cross-attention layer of the pre-trained diffusion model. q The calculated query matrix, K c =c c W k It is composed of content feature c c The key weights W of the cross-attention layer of the pre-trained diffusion model k The calculated key matrix, V c =c c W v It is composed of content feature c c The value weight W of the cross-attention layer of the pre-trained diffusion model v The calculated value matrix;

[0039] The style features of the style image inserted through the new cross-attention layer are expressed as follows:

[0040] Z s =Attention(Q,K) s V s ),

[0041] Where Q = ZW q It consists of query features Z and query weights W from the cross-attention layer of the pre-trained diffusion model. q The calculated query matrix, K s =c s W k ′ is composed of style feature c s And the newly added trainable key weights W k The calculated key matrix, V s =c s W v ′ is composed of style feature c s And the newly added trainable value weights W v The calculated value matrix;

[0042] The outputs of the original cross-attention layer and the new cross-attention layer are added together to combine content and style features. The expression is: Z new =Z c +Z s .

[0043] Step 2: Construct a training set that covers multiple data types.

[0044] To improve the performance of the image style transfer model by constructing a training set covering multiple data types, images of various style types are collected, and two types of image pairs (content image, style image) are constructed. One type of image pair is obtained by copying an image, and the other type of image pair consists of a content image and a style image that has the same style as the content image but different content.

[0045] Step 3: Train the image style transfer model using the training set and loss function.

[0046] The image style transfer model is trained using a training set and a loss function, including: inputting a content image into a content extraction module, which extracts a content representation of the content image; inputting a style image into a style extraction module, which extracts a style representation of the style image; inputting the style representation into a projection network, which maps the style representation to style features aligned with the text feature dimensions in the pre-trained diffusion model; inputting the content image into a VAE encoder to obtain an image representation of the content image in the latent space, and randomly adding noise to the image representation of the content image to obtain initial latent noise for the subsequent diffusion process; inputting the content representation, style features, and initial latent noise into the diffusion model of the style transfer module, where the model predicts the noise; calculating the loss using the same loss function as the original diffusion model; and optimizing the trainable parameters of the image style transfer model through backpropagation based on the loss.

[0047] Furthermore, the trainable parameters of the image style transfer model include all parameters of the projection network and the key weights W of the new cross-attention layer added in the style transfer module. k Sum of weights W v The parameters of other modules of the model are the same as those of the pre-trained model and do not need to be updated.

[0048] Step 4: Based on the trained image style transfer model, input the content image and style image to generate a stylized result image with the target content and target style.

[0049] Appendix Figure 5 This is a schematic diagram of the overall network structure of a decoupled image style transfer method based on a diffusion model proposed in this invention, as shown below. Figure 5As shown, a stylized result image with target content and target style is generated based on a trained image style transfer model. The specific process is as follows: the content image is input into the content extraction module to extract the content representation of the content image; the style image is input into the style extraction module to extract the style representation of the style image; the style representation is input into the trained projection network to obtain style features aligned with the text feature dimensions in the pre-trained diffusion model; the content image is input into the VAE encoder to obtain the image representation of the content image in the latent space, and specific noise addition processing is performed on the image representation of the content image to obtain the initial latent noise for the subsequent diffusion process; the content representation, style features, and initial latent noise are input into the trained style transfer module to obtain style-transferred features; the style-transferred features are decoded to obtain the style-transferred image, wherein the content of the style-transferred image is consistent with the content image, and the style is consistent with the style image.

[0050] In this invention, the images used during network training are from the WikiArt dataset, including styles such as Abstract Expressionism, Impressionism, Primitivism, Neo-Impressionism, Neoclassicism, and Post-Impressionism. The images are in JPG format, and all image sizes are adjusted to 512*512 using the `resize` function of the `Image` module in Python's PIL library. The brightness transformation method used to remove image colors in the content extraction module is IUR 601-2, specifically the formula Y = 0.299*R + 0.587*G + 0.114*B, where R represents the red component, G represents the green component, and B represents the blue component. The diffusion model used in the content extraction module is the ADM diffusion model pre-trained on ImageNet. Since the images in the ImageNet dataset are all photographs, the ADM diffusion model can convert input images from different domains into a latent domain through a forward process, and then convert them back into a photographic domain through a reverse process, eliminating stylistic features and leaving only the image content. Both the forward denoising and reverse denoising processes in the content extraction module employ DDIM sampling technology, with a maximum time step T of 600 and a sample number S of 40. The style extraction module and the CLIP image encoder in the overall network framework are both encoders from OpenCLIP ViT-H / 14. The diffusion model used in the style transfer process is SDv1.5. The style transfer module only introduces a new cross-attention layer in the part of the diffusion model that captures style information, corresponding to the third layer of the downsampling stage, the intermediate block, and the second layer of the upsampling stage in the SDv1.5 model. During training, the AdamW optimizer is used to update the corresponding weight gradients, with a fixed learning rate of 0.0001 and a weight decay of 0.01.

[0051] It should be noted that the above embodiments are only used to further illustrate the present invention and should not be construed as limiting the scope of protection of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to these embodiments without departing from the principles of the present invention, which still fall within the scope of protection of the present invention.

Claims

1. A decoupled image style transfer method based on a diffusion model, characterized in that, include: Construct an image style transfer model based on content and style decoupling, wherein the image style transfer model includes at least a content extraction module, a style extraction module, a projection network, and a style transfer module; Construct a training set that covers multiple data types; The image style transfer model is trained using a training set and a loss function; Based on a trained image style transfer model, input content image and style image, generate stylized result image with target content and target style; The content extraction module is used to extract content information from the image, and the specific process is as follows: a. Given an input image, remove its color using a brightness transformation method to obtain a grayscale image of the image; b. Use a pre-trained diffusion model to perform forward noise addition and reverse noise reduction on the grayscale image to remove brushstroke and texture style details and extract the image content. The style extraction module is used to extract style representations from style images. The specific process is as follows: a. Given an input style image, extract the content of the style image using the content extraction module; b. Input the original style image into the CLIP image encoder to obtain a global representation containing style and content. Input the content of the style image into the CLIP image encoder to obtain a content representation of the style image. Subtract the content representation from the global representation to obtain the style representation, which is expressed as: f s =E(I s ), Among them, I s Representing style images, The content of the style image is represented by E, which represents the CLIP image encoder, and f is the image encoder. s A global representation of the style image. The content representation of the style image, Style representation of a style image; The style transfer module is used to perform style transfer based on the content features of the content image and the style features of the style image. It adds a new cross-attention layer to the module that captures style information in the diffusion model. Combining the original cross-attention layer and the newly added cross-attention layer, the content features and style features are injected into the denoising process of the diffusion model. The content features of the content image are inserted through the original cross-attention layer of the pre-trained diffusion model, and its expression is: Z c =Attention(Q,K c ,V c ), Where Q = ZW q It consists of query features Z and query weights W from the cross-attention layer of the pre-trained diffusion model. q The calculated query matrix, K c =c c W k It is composed of content feature c c The key weights W of the cross-attention layer of the pre-trained diffusion model k The calculated key matrix, V c =c c W v It is composed of content feature c c The value weight W of the cross-attention layer of the pre-trained diffusion model v The calculated value matrix; Style features of the style image are inserted through a new cross-attention layer, expressed as: Z s =Attention(Q,K s ,V s ), Where Q = ZW q It consists of query features Z and query weights W from the cross-attention layer of the pre-trained diffusion model. q The calculated query matrix, K s =c s W' k It is composed of style features c s And the newly added trainable key weights W' k The calculated key matrix, V s =c s W' v It is composed of style features c s And the newly added trainable value weights W' v The calculated value matrix; The outputs of the original cross-attention layer and the new cross-attention layer are added together to combine content and style features. The expression is: Z new =Z c +Z s ; The training of the image style transfer model using a training set and a loss function includes: a. Input the content image into the content extraction module, and the content extraction module extracts the content representation of the content image; b. Input the style image into the style extraction module, and the style extraction module extracts the style representation of the style image; c. Input the style representation into the projection network, which maps the style representation to style features aligned with the text feature dimensions in the pre-trained diffusion model; d. Input the content image into the VAE encoder to obtain the image representation of the content image in the latent space, and perform random noise addition processing on the image representation of the content image to obtain the initial latent noise for the subsequent diffusion process; e. Input the content representation, style features, and initial potential noise into the diffusion model of the style transfer module, and the model predicts the noise; f. Calculate the loss using the same loss function as the original diffusion model; g. Based on the loss, optimize the trainable parameters of the image style transfer model through backpropagation.

2. The decoupled image style transfer method based on a diffusion model according to claim 1, characterized in that, The projection network is used to project style representations into style features aligned with the text feature dimensions in the pre-trained diffusion model. The projection network consists of a linear layer and a normalization layer. The linear layer is used to align dimensions, and the normalization layer is used to improve training stability and convergence speed.

3. The decoupled image style transfer method based on a diffusion model according to claim 1, characterized in that, The training parameters of the image style transfer model include all parameters of the projection network and the key weights W of the new cross-attention layer added in the style transfer module. ' k Sum weight W ' v The parameters of other modules of the model are the same as those of the pre-trained model and do not need to be updated.

4. The decoupled image style transfer method based on a diffusion model according to claim 1, characterized in that, The process of generating a stylized image with target content and target style based on a trained image style transfer model is as follows: a. Input the content image into the content extraction module to extract the content representation of the content image; b. Input the style image into the style extraction module to extract the style representation of the style image; c. Input the style representation into the trained projection network to obtain style features that are aligned with the text feature dimensions in the pre-trained diffusion model; d. Input the content image into the VAE encoder to obtain the image representation of the content image in the latent space, and perform specific noise addition processing on the image representation of the content image to obtain the initial latent noise for the subsequent diffusion process; e. Input the content representation, style features, and initial latent noise into the trained style transfer module to obtain the style-transferred features; f. Decode the style-transferred features to obtain a style-transferred image, wherein the content of the style-transferred image is consistent with the content image, and the style is consistent with the style image.

Images