Untrained image style transfer method and device based on diffusion model and medium

Abstract

The application discloses a kind of training-free image style migration method, device and medium based on diffusion model, the method includes: based on the text embedding corresponding to the style image generated in the multi-modal BLIP-2 encoder in BLIP-Diffusion;Based on the residual module and self-attention module of U-Net, the key intermediate feature is extracted from content branch and style branch by denoising diffusion implicit model DDIM inversion technology;The intermediate feature is used as the spatial feature to be replaced in the target branch;According to the different needs of content and style, selectively replace the specific layer of residual block and self-attention module in content injection, replace the specific layer of self-attention module in style injection, realize the optimal fusion of content and style;Using the step-by-step characteristics of diffusion model, content injection and style injection in target branch are carried out separately.The application adopts training-free mode, realizes the balance between content preservation and style injection, realizes high-quality image style migration.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of diffusion models and image generation, and particularly relates to a diffusion model-based image style transfer method and device and medium. BACKGROUND

[0002] Each image has its specific content and unique style. When we are interested in a specific style and want to turn our photos into new images combined with this style, it is called style transfer, which is a technique of injecting the style of one image into the content of another image. The main goal of style transfer is to inject the artistic features of the style image into the content image without destroying the essence of the content structure, which is not a simple task and has been widely studied. How to balance between content preservation and style injection is still a challenge. Existing methods cannot produce balanced stylized images, which are either too artistic and deviate from the original content, or too content-oriented and lack style features. In addition, the target artistic style is often very subtle, making it difficult for well-trained networks to master them well. These limitations severely affect the quality of stylization.

[0003] Recently, text-to-image diffusion models with text and image feature interaction capabilities provide multi-modal capabilities for various visual tasks. Pre-trained diffusion models are used to learn text embeddings of style images or design objective functions to fine-tune denoising U-Net.

[0004] The present application combines the text features and spatial features of style images to more accurately guide the stylization process. Through this combination, the present application can more effectively control the direction and degree of style transfer, ensuring that the generated images achieve the desired effect in both content and style. At the same time, to adapt to the different needs of content and style, the present application designs different injection modules that can ensure effective fusion of style features and content features in the target branch. SUMMARY

[0005] The present application provides a diffusion model-based training-free image style transfer method, device and medium, which makes the diffusion model-based stylization result output both preserve the structural features of the content image and retain the style features of the style image, achieving a balance between content preservation and style injection, and realizing high-quality style transfer.

[0006] Technical solution: the diffusion model-based training-free image style transfer method provided by the present application comprises the following steps:

[0007] (1) generating a text embedding corresponding to a style image based on the multi-modal BLIP-2 encoder in BLIP-Diffusion;

[0008] (2) Based on the residual module and the self-attention module of the U-Net, the key intermediate features are extracted from the content branch and the style branch through the denoising diffusion implicit model DDIM inversion technology; the intermediate features are used as the spatial features to be replaced in the target branch;

[0009] (3) According to different requirements of content and style, the specific layers of the residual block and the self-attention module are selectively replaced in the content injection, and the specific layers of the self-attention module are replaced in the style injection, so as to realize the optimal fusion of content and style;

[0010] (4) The content injection and the style injection in the target branch are carried out separately by using the step-by-step characteristics of the diffusion model.

[0011] Further, the step (1) is implemented as follows:

[0012] The pre-trained BLIP-2 encoder is represented as F to convert the visual features of the style image into a text-aligned representation; the text embedding of the CLIP text encoder ψ is retained in the diffusion model to enhance the expression of content information; the above two text embeddings are combined through a connection operation to guide the subsequent style transfer process:

[0013]

[0014] Where P c is an optional guiding text, I s is a style image.

[0015] Further, the extraction process of the intermediate features in step (2) is as follows:

[0016] Each layer of the pre-trained diffusion U-Net is composed of a residual block, a self-attention module for enhancing representation, and a cross-attention module for interacting with the text condition; for step t of the lth layer in the U-Net, the residual block with intermediate features outputs The image based on the self-attention module enhancement is:

[0017]

[0018] Where, is the final output of the self-attention, is the mapping of the output of the residual block.

[0019] Further, the different self-attention module replacement strategies in step (3) are as follows:

[0020] In the content injection, the residual feature in the target branch is replaced by the residual feature in the content branch.The query Q and the key K in the self-attention module are also replaced in step t:

[0021]

[0022] wherein, is the projection of the content branch , is the projection of the target branch ;

[0023] In style injection, similar to content injection, style space features are injected into the target branch in step t, replacing the key K and value V elements:

[0024]

[0025] wherein, is the projection of the style branch , is the projection of the target branch .

[0026] Further, the step (3) described in the content injection selectively replaces the specific layers of the residual block and the self-attention module to replace the 3rd to 8th layers in the residual block and the 4th to 11th layers in the self-attention module.

[0027] Further, the step (3) described in the style injection replaces the specific layers of the self-attention module to replace the 4th to 11th layers in the self-attention module.

[0028] Further, the step (4) described in the target branch realizes step-by-step injection as follows:

[0029] Content injection is performed in the early stage of the reverse process of the pre-trained diffusion model, and style injection is performed in the later stage; and an hyperparameter alpha is introduced to calculate the time separation node of different injections The formula for separate injection in the target branch is as follows:

[0030]

[0031] wherein, t is in (0, T]; T is the total number of steps of the diffusion model.

[0032] Further, the value of the hyperparameter alpha is 0.2.

[0033] The device of the application comprises a memory and a processor, wherein:

[0034] The memory is used to store a computer program capable of running on the processor;

[0035] A processor configured to execute the steps of the diffusion model based training-free image style transfer method as described above when running the computer program.

[0036] A storage medium storing a computer program which, when executed by at least one processor, implements the steps of the diffusion model based training-free image style transfer method as described above.

[0037] Advantages: Compared with the prior art, the advantages of the present application are: the present application uses the text features extracted by the BLIP-2 encoder and the spatial features obtained by the denoising diffusion implicit model DDIM as style guidance, which significantly improves the style matching degree in the stylization process; this combination not only enhances the accuracy of style transfer, but also improves the artistic expressiveness of the final image; the present application designs two different self-attention replacement strategies, which optimize the preservation of structural information and high-level information by replacing the query Q, key K and value V according to the features of content and style, and this differentiated processing method makes the style transfer result more in line with expectations; the present application focuses on the preservation of content information, and replaces the residual features to ensure that the basic structure of the content image is not destroyed; in order to control the degree of stylization, the proportion of content and style space injection is controlled by using hyperparameters, so as to realize accurate control of the degree of stylization; this control mechanism provides users with more flexibility to adjust the effect of style transfer according to specific needs; the present application adopts a training-free method, which avoids resource consumption and time cost in the traditional training process; this method enables the diffusion model based image generation model to quickly output results that meet the requirements of style transfer, realizing controllable and higher quality image style transfer. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 Fig. 1 is a schematic diagram of the overall framework of the diffusion model based training-free image style transfer;

[0039] Figure 2 Fig. 4 is a schematic diagram of content injection in the network architecture of the present application;

[0040] Figure 3 Fig. 5 is a schematic diagram of style injection in the network architecture of the present application;

[0041] Figure 4 Fig. 6 is a schematic diagram of the results of image style transfer using different network models; (a) is a schematic diagram of the results of image style transfer of InST; (b) is a schematic diagram of the results of image style transfer of DiffuseIT; (c) is a schematic diagram of the results of image style transfer of CAP-VSTNet; (d) is a schematic diagram of the results of image style transfer of the present application. DETAILED DESCRIPTION

[0042] The application will be described in further detail below with reference to the drawings.

[0043] The application provides a training-free image style transfer method based on a diffusion model. First, a multi-modal BLIP-2 encoder from BLIP-Diffusion is used to generate a text embedding corresponding to a style image as a text embedding of a target branch; second, special attention is paid to the residual module and the self-attention module of the U-Net, and the denoising diffusion implicit model (DDIM) inversion technology is used to extract intermediate features from the content and style branches as spatial features to be replaced in the target branch; then, in view of the characteristics that the content needs to keep the structure and the style needs to show artistic visual effects, two different replacement strategies of the self-attention module are designed; finally, in order to generate a stylized image with balanced content and style, the step-by-step characteristics of the diffusion model are used to inject the content and the style in the target branch separately, starting from the noisy content image, and the denoising process follows the denoising diffusion implicit model DDIM, so as to obtain the corresponding stylized image, as shown in Figure 1 The method comprises the following steps:

[0044] Step 1: Since the artistic style in the image style task is likely to be abstract and not easy to describe by text, the BLIP-2 encoder F in BLIP-Diffusion is adopted to convert the visual features of the style image into a text-aligned representation, and the natural text prompt is used to enhance the content information:

[0045]

[0046] where P c is an optional guiding text, I s is a style image, and the empty text "" is used as the input text of the BLIP-2 encoder to help capture global and abstract style representations to meet the needs of the image style transfer task.

[0047] In addition, the application can be extended to a text-guided editing task, in which case the text input condition becomes:

[0048]

[0049] where P edit is a user-specified text prompt that allows changes in the content structure in the final stylized image, such as changing a cat into a dog.

[0050] Step 2: Each layer of the pre-trained diffusion U-Net is composed of a residual block, an enhanced self-attention module, and a cross-attention module that interacts with the text condition. In this work, we focus on the first two blocks. For step t at layer l, with intermediate features the residual block outputs The self-attention module enhances the image representation: where is the final output of the self-attention, is a mapping of the output of the residual block.

[0051] Step 3: Different injection modules are designed for different features of content and style images.

[0052] For traditional diffusion model-based works, simply adding noise to the content image as the starting image of the U-Net can cause deviation of content structure information. Our invention injects spatial features and targets the differences between content and style information. Content information mainly retains the global and structural information of the content image, while style information mainly targets some abstract information. Based on the above features, content injection as shown in Figure 2 is proposed. In the content information injection, the residual feature in the target branch is replaced by the residual feature in the content branch. The query Q and key K in the self-attention module are also replaced at step t:

[0053]

[0054] where is the projection of the in the content branch, is the projection of the in the target branch. Note that the replacement of the graph ensures the injection of spatial layout and structural information in the content image, while replacing the residual feature in the target branch with the in the content branch, where l' is the layer selected for residual replacement, which is more conducive to the preservation of content structure.

[0055] Style injection as shown in Figure 3 is similar to content injection. At step t, style spatial features are injected into the target branch, replacing the key K and value V elements:

[0056]

[0057] where is the projection of the in the style branch, is the projection of the The text information and spatial information of the style image are fused, and unlike the replacement of the key K and value V in the cross attention in the traditional work, the replacement of the key K and value V in the self-attention is performed in the present application, so that the style information can be better injected, and the representation of the style information is more accurate.

[0058] The present application injects content and style at different stages of the target branch. Specifically, by setting the hyperparameter a to 0.2, it means that in the first 80% of the style transfer process, the injection of content information is mainly performed, and in the subsequent 20% stage, the injection of style information is focused on. This phased processing strategy aims to ensure the gradual fusion of content and style, so as to realize the harmony and unity of content and style in the final image. In the content injection stage, the method of the present application focuses on replacing the 3rd to 8th layers in the residual block and the 4th to 11th layers in the self-attention module. This replacement operation is to more accurately inject the features of the content image in the generation process of the image, ensuring clear expression of content information in the final image. In the style injection stage, the method of the present application focuses on replacing the 4th to 11th layers in the self-attention module. The purpose of this step is to inject the features of the style image into the generation process, so as to give the image new visual effects and style while maintaining the integrity of the content.

[0059] Step 4: Separate injection in the target branch. Empirically, the early stages of the inverse process of the pre-trained diffusion model affect the overall structure and spatial layout, while the later stages focus on more detailed visual effects. Therefore, content injection is performed in the early stage, and style injection is performed in the later stage. A hyperparameter a is introduced, and the time separation node of different injections is calculated T is the total number of steps of the diffusion model. The formula for separate injection in the target branch is as follows:

[0060]

[0061] Where t is in (0, T].

[0062] The method proposed in the present application uses the pre-trained BLIP-2 encoder and the text encoder of CLIP, and the latent diffusion model Stable Diffusion as the diffusion model to achieve efficient and accurate image style conversion. In this invention, the input content image and style image are processed through specific processes to generate a new image with the target style.

[0063] The present application also provides a device apparatus comprising a memory and a processor, wherein: the memory is used to store a computer program capable of running on the processor; the processor is used to execute the steps of the training-free image style transfer method based on the diffusion model as described above when running the computer program.

[0064] The application also provides a storage medium storing a computer program, which, when executed by at least one processor, implements the steps of the training-free image style transfer method based on a diffusion model.

[0065] In the present application, once the corresponding text information and spatial information are extracted from the content image and the style image, a series of steps are taken to ensure the integrity of the content and the accuracy of the style during the image style transfer process. Specifically, the present application involves using the final result obtained by denoising diffusion implicit model DDIM inversion in the content branch as the initial noise of the target branch, which serves as the starting point of the style transfer process. Such a processing method helps to maintain the basic features of the content image during the style transfer process, thereby improving the content integrity of the final generated image. Through the above steps, the method of the present application can effectively balance the preservation of content and the conversion of style during the image style transfer process, generating an image that is faithful to the original content and has new style characteristics.

[0066] Specifically, the method of the present application first receives a content image, such as a picture of a bear, and a style image, such as a picture of a flame. In addition, an editable text input, such as "dog", is also received. In the processing flow, first, the denoising diffusion implicit model DDIM inversion technology is used to add noise to the content image and the style image, generating a noisy image. The noisy image is then used as the input of the U-Net network for the subsequent style transfer process. It is worth noting that the input of the U-Net network is strictly provided by the noisy image obtained by denoising diffusion implicit model DDIM inversion, rather than the output of the U-Net in the previous step, to ensure the accuracy of the information and the continuity of the process.

[0067] In the text input processing of the target branch, the method of the present application encodes the style image into the BLIP-2 encoder to extract the text information aligned with the style image, such as "flame". Then, the extracted text information is combined with the specified text. In general cases, the specified text is empty to better learn the features of the style image. However, when it comes to image editing tasks, such as converting the structure of "bear" to that of "dog", the specified text is set to "dog" to guide the structure conversion of the image. Finally, the combined text information is used as the text embedding in the target branch, thereby achieving precise control of the image content and style during the style transfer process.

[0068] In order to verify the effectiveness of the present application, comparative tests were carried out respectively with the use of other three image style transfer models. The three network models are the current popular network models InST (Inversion-based style transfer with diffusion models), DiffuseIT (Diffusion-based image translation using disentangled style and content representation) and CAP-VSTNet (CAP-VSTNet: content affinity preserved versatile style transfer.). All methods are used for style transfer on the same size content image and style image. As shown in Figs. 1 to 4, the generation results of (a) to (d) are respectively the generation results of InST, DiffuseIT, CAP-VSTNet and the algorithm proposed in the present application. The present application can make the output of the image style transfer model based on the diffusion model more harmonious with smaller calculation and time cost. Both the content structure of the original content image and the style of the style image are preserved, and the balance between content and style is truly realized. Figure 4

[0069] The above description is only an embodiment of the present application, and does not limit the protection scope of the present application. Any equivalent transformation or direct or indirect application in other related technical fields based on the content of the specification and drawings of the present application should be included in the protection scope of the present application.​

Claims

1. A diffusion model based training-free image style transfer method, characterized in that, The method comprises the following steps: (1) generating a text embedding corresponding to a style image based on a multi-modal BLIP-2 encoder in BLIP-Diffusion; (2) extracting key intermediate features from the content branch and the style branch through a denoising diffusion implicit model (DDIM) inversion technology based on a U-Net residual module and a self-attention module; the intermediate features are used as spatial features to be replaced in the target branch; (3) selectively replacing specific layers of the residual block and the self-attention module in the content injection and replacing specific layers of the self-attention module in the style injection according to different requirements of the content and the style to achieve optimal fusion of the content and the style; (4) separating the content injection and the style injection in the target branch by using the step-by-step characteristics of the diffusion model; The implementation process of step (1) is as follows: The pre-trained BLIP-2 encoder is denoted as F to convert the visual features of the style image into a text-aligned representation; the text embedding of the CLIP text encoder ψ is retained in the diffusion model to enhance the expression of content information; the two kinds of text embeddings are combined through a connection operation to guide the subsequent style transfer process: where P c is an optional guidance text, I s is a style image; The extraction process of the intermediate features in step (2) is as follows: Each layer of the pre-trained diffusion U-Net is composed of a residual block, a self-attention module that enhances the representation, and a cross-attention module that interacts with the text condition; for step t of the l-th layer in the U-Net, the residual block with intermediate features outputs the image enhanced based on the self-attention module as: wherein, is the final output of the self-attention, is a mapping of the output of the residual block; The different self-attention module replacement strategies in step (3) are as follows: In content injection, the residual features in the target branch are replaced by the residual features in the content branch The query Q, key K in the self-attention module are also replaced in step t: wherein is a content branch of projected, is a target branch of projected; In the style injection, similar to the content injection, the style spatial features are injected into the target branch at step t to replace the key K and value V elements: wherein, the style branch of projecting, the target branch of projecting; Step (4) realizes step-by-step injection in the target branch as follows: Content injection is performed in the early stage of the reverse process of pre-training diffusion model, and style injection is performed in the later stage; and an over-parameter α is introduced to calculate the time separation node of different injections The formula for separate injection at the target branch is as follows: Wherein, t∈(0,T];T is the total number of steps of the diffusion model.

2. The diffusion model based training-free image style transfer method according to claim 1, characterized in that, The specific layers of the residual block and the self-attention module selectively replaced in the content injection in step (3) are the 3rd to 8th layers in the residual block and the 4th to 11th layers in the self-attention module.

3. The diffusion model based training-free image style transfer method according to claim 1, wherein, The specific layers of the self-attention module replaced in the style injection in step (3) are the 4th to 11th layers in the self-attention module.

4. The diffusion model based training-free image style transfer method according to claim 1, characterized in that, The value of the hyperparameter α is 0.

2.

5. An apparatus device comprising: It comprises a memory and a processor, wherein: The memory is used to store a computer program capable of running on the processor; The processor is used to execute the steps of the diffusion model-based training-free image style transfer method according to any one of claims 1 to 4 when running the computer program.

6. A storage medium, characterized by The storage medium stores a computer program, and the computer program is executed by at least one processor to implement the steps of the diffusion model-based training-free image style transfer method according to any one of claims 1 to 4.

Images