Training-free image style migration method and device based on diffusion model and medium

Abstract

The invention discloses a training-free image style migration method and device based on a diffusion model and a medium. The method comprises the steps that text embedding corresponding to a style image is generated based on a multi-mode BLIP-2 encoder in BLIP-Diffusion; based on a U-Net residual module and a self-attention module, key intermediate features are extracted from a content branch and a style branch through a de-noising diffusion implicit model DDIM inversion technology; the intermediate feature is used as a spatial feature to be replaced in a target branch; according to different requirements of contents and styles, selectively replacing a residual block and a specific layer of a self-attention module in content injection, and replacing the specific layer of the self-attention module in style injection to realize optimal fusion of the contents and the styles; and content injection and style injection in the target branch are performed separately by using the gradual characteristics of the diffusion model. According to the method, a training-free mode is adopted, balance between content storage and style injection is achieved, and high-quality image style migration is achieved.

Description

Technical Field

[0001] The present invention relates to the field of diffusion models and image generation, and in particular to an image style migration method, device and medium based on a diffusion model. Background Art

[0002] Every image has its specific content and unique style. When we are interested in a specific style and want to turn our own photos into new images combined with this style, this is called style transfer, a technique that injects the style of one image into the content of another image. The main goal of style transfer is to inject the artistic features of the style image into the content image without destroying the essential structure of the content. This is not a simple task and has been widely studied. How to find a balance between content preservation and style injection remains a challenge. Existing methods cannot produce balanced stylized images, which are manifested as too many artistic features that deviate from the original content; or too strong content features and insufficient style features. In addition, the target artistic styles are often very subtle, making it difficult for well-trained networks to grasp them well. These limitations seriously affect the quality of stylization.

[0003] Recently, text-to-image diffusion models with the ability to interact with image features have provided multimodal capabilities for various vision tasks. Pre-trained diffusion models are used to learn text embeddings for style images or to design objective functions to fine-tune denoising U-Nets.

[0004] The present invention combines the text features and spatial features of the style image to guide the stylization process more accurately. Through this combination, the present invention can more effectively control the direction and degree of style migration, ensuring that the generated image achieves the desired effect in terms of content and style. At the same time, in order to meet the needs of different natures of content and style, the present invention designs different injection modules, which can ensure that the style features and content features are effectively integrated in the target branch. Summary of the invention

[0005] Purpose of the invention: The present invention provides a method, device and medium for training-free image style transfer based on a diffusion model, so that the stylized result output based on the diffusion model retains both the structural features of the content image and the style features of the style image, achieves a balance between content preservation and style injection, and realizes high-quality style transfer.

[0006] Technical solution: The present invention provides a method for training-free image style transfer based on a diffusion model, comprising the following steps:

[0007] (1) Generate text embedding corresponding to the style image based on the multimodal BLIP-2 encoder in BLIP-Diffusion;

[0008] (2) Based on the residual module and self-attention module of U-Net, the key intermediate features are extracted from the content branch and the style branch through the denoising diffusion implicit model DDIM inversion technology; the intermediate features are used as the spatial features to be replaced in the target branch;

[0009] (3) According to the different requirements of content and style, we selectively replace specific layers of the residual block and the self-attention module in content injection, and replace specific layers of the self-attention module in style injection to achieve the optimal fusion of content and style;

[0010] (4) Utilizing the gradual characteristics of the diffusion model, the content injection and style injection in the target branch are performed separately.

[0011] Furthermore, the implementation process of step (1) is as follows:

[0012] The pre-trained BLIP-2 encoder is denoted as F to convert the visual features of the style image into a representation aligned with the text; the text embedding of the CLIP text encoder ψ is retained in the diffusion model to enhance the expression of content information; the above two text embeddings are combined through a concatenation operation to guide the subsequent style transfer process:

[0013]

[0014] Among them, P c is an optional instructional text. s is the style image.

[0015] Furthermore, the extraction process of the intermediate features in step (2) is as follows:

[0016] Each layer of the pre-trained diffusion U-Net consists of a residual block, a self-attention module for enhanced representation, and a cross-attention module for interaction with text conditions; for step t of the first layer in U-Net, with intermediate features The residual block will output The image enhanced based on the self-attention module is:

[0017]

[0018] in, is the final output of self-attention, is the mapping of the output of the residual block.

[0019] Furthermore, the different self-attention module replacement strategies in step (3) are:

[0020] In content injection, the residual features in the target branch are injected Replaced with residual features in the content branch The query Q and key K in the self-attention module are also replaced in step t:

[0021]

[0022] in, It is a content branch Projected, The target branch projected

[0023] In style injection, similar to content injection, in step t, the style space features are injected into the target branch, replacing the key K and value V elements:

[0024]

[0025] in, Style branch Projected, The target branch Projected.

[0026] Furthermore, in step (3), the specific layers of the residual block and the self-attention module are selectively replaced during content injection by replacing the 3rd to 8th layers in the residual block and the 4th to 11th layers in the self-attention module.

[0027] Furthermore, in step (3), replacing the specific layer of the self-attention module in the style injection is replacing the 4th to 11th layers in the self-attention module.

[0028] Furthermore, step (4) implements the step-by-step injection in the target branch as follows:

[0029] Content injection is performed in the early stage of the reverse process of the pre-trained diffusion model, and style injection is performed in the later stage; and a hyperparameter α is introduced to calculate the time separation nodes of different injections The formula for split injection in the target branch is as follows:

[0030]

[0031] Where t∈(0,T]; T is the total number of steps in the diffusion model.

[0032] Furthermore, according to the diffusion model-based training-free image style transfer method of claim 7, it is characterized in that the value of the hyperparameter α is 0.2.

[0033] A device according to the present invention includes a memory and a processor, wherein:

[0034] A memory for storing computer programs that can be run on the processor;

[0035] The processor is used to execute the steps of the above-mentioned diffusion model-based training-free image style transfer method when running the computer program.

[0036] A storage medium described in the present invention stores a computer program, and when the computer program is executed by at least one processor, the steps of the above-mentioned diffusion model-based training-free image style transfer method are implemented.

[0037] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects: the present invention utilizes the text features extracted by the BLIP-2 encoder and the spatial features obtained by the denoising diffusion implicit model DDIM as style guidance, thereby significantly improving the style matching degree in the stylization process; this combination not only enhances the accuracy of style transfer, but also enhances the artistic expression of the final image; the present invention designs two different self-attention replacement strategies, and optimizes the retention of structural information and high-level information by replacing the query Q, key K and value V for the features of content and style, and this differentiated processing method makes the style transfer result more in line with expectations; the present invention focuses on the retention of content information, and ensures that the basic structure of the content image is not destroyed by replacing the residual features; in order to control the degree of stylization, hyperparameters are used to control the ratio of content and style space injection, thereby achieving precise control of the degree of stylization; this control mechanism provides users with more flexibility to adjust the effect of style transfer according to specific needs; the present invention adopts a training-free method, which avoids the resource consumption and time cost in the traditional training process; this method enables the image generation model based on the diffusion model to quickly output results that meet the requirements of style transfer, and realizes controllable and higher-quality image style transfer. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of the overall framework of training-free image style transfer based on the diffusion model;

[0039] Figure 2 A schematic diagram of content injection in the network architecture of the present invention;

[0040] Figure 3 A schematic diagram of style injection in the network architecture of the present invention;

[0041] Figure 4 Schematic diagrams of image style transfer results using different network models; among them, (a) is a schematic diagram of the image style transfer results of InST; (b) is a schematic diagram of the image style transfer results of DiffuseIT; (c) is a schematic diagram of the image style transfer results of CAP-VSTNet; (d) is a schematic diagram of the image style transfer results of the present invention. DETAILED DESCRIPTION

[0042] The present invention is further described in detail below with reference to the accompanying drawings.

[0043] The present invention provides a training-free image style transfer method based on a diffusion model. First, a multimodal BLIP-2 encoder from BLIP-Diffusion is used to generate text embeddings corresponding to a style image as text embeddings of a target branch; second, special attention is paid to the residual module and self-attention module of U-Net, and the denoising diffusion implicit models (DDIM) inversion technology is used to extract intermediate features from content and style branches as spatial features to be replaced in the target branch; then, in view of the characteristics that content needs to maintain structure and style needs to show artistic visual effects, two different replacement strategies of self-attention modules are designed; finally, in order to generate a stylized image with balanced content and style, the gradual characteristics of the diffusion model are used to separate the content and style injections in the target branch, and starting from the noisy content image, the denoising process follows the denoising diffusion implicit model DDIM, thereby obtaining the corresponding stylized image, such as Figure 1 As shown, the following steps are included:

[0044] Step 1: Since the artistic style in the image style task is likely to be abstract and not easy to describe through text, the BLIP-2 encoder F in BLIP-Diffusion is used to convert the visual features of the style image into a text-aligned representation, while enhancing the content information with natural text cues:

[0045]

[0046] Among them, P c is an optional instructional text. s is a style image, using empty text “” as the input text of the BLIP-2 encoder to help capture global and abstract style representations to meet the needs of image style transfer tasks.

[0047] Furthermore, the present invention can be extended to text-guided editing tasks, in which case the text input condition becomes:

[0048]

[0049] Among them, edit It is a user-specified textual hint that allows to change the content structure in the final stylized image, such as changing a cat into a dog.

[0050] Step 2: Each layer of the pre-trained diffusion U-Net consists of a residual block, a self-attention module for enhanced representation, and a cross-attention module for interaction with text conditions. In this work, we focus on the first two blocks in particular. For step t at layer 1, with intermediate features The residual block will output The self-attention module enhances the image representation: in is the final output of self-attention, is the mapping of the output of the residual block.

[0051] Step 3: Design different injection modules according to the different characteristics of content and style images.

[0052] For traditional diffusion model-based work, the content image is simply noised as the starting image of U-Net. This method will cause deviation of content structure information. However, our invention injects spatial features and targets the difference between content information and style information. The content information mainly retains the global and structural information of the content image, while the style image mainly targets some abstract information. Based on the above features, we propose the following Figure 2 In the content information injection, the residual features in the target branch are injected Replaced with residual features in the content branch The query Q in the self-attention module, the key K is also replaced in step t:

[0053]

[0054] in, It is a content branch Projected, The target branch The replacement of the attention map ensures the injection of spatial layout and structural information in the content image, while the residual features in the target branch are Replace with the content branch Where l' is the layer selected for residual replacement, which is more conducive to preserving the content structure.

[0055] like Figure 3 The style injection shown is similar to the content injection. In step t, the style space features are injected into the target branch, replacing the key K and value V elements:

[0056]

[0057] in, Style branch Projected, The target branch The present invention fuses the text information of the style image with the spatial information. Unlike the traditional work that replaces the key K and value V in the cross attention, our invention replaces the key K and value V in the self-attention, which can better inject style information and make the representation of style information more accurate.

[0058] The present invention performs content and style injection respectively at different stages of the target branch. Specifically, by setting the hyperparameter α to 0.2, it means that the content information is mainly injected in the first 80% stage of the style migration process, while the subsequent 20% stage focuses on the injection of style information. This staged processing strategy aims to ensure the gradual fusion of content and style, so as to achieve the harmonious unity of content and style in the final image. In the content injection stage, the method of the present invention pays special attention to replacing the 3rd to 8th layers in the residual block and the 4th to 11th layers in the self-attention module. This replacement operation is to more accurately inject the features of the content image in the image generation process, ensuring the clear expression of the content information in the final image. In the style injection stage, the method of the present invention focuses on replacing the 4th to 11th layers in the self-attention module. The purpose of this step is to inject the features of the style image into the generation process, thereby giving the image a new visual effect and style while maintaining the integrity of the content.

[0059] Step 4: Separate injection in the target branch. Empirically, the early stage of the inverse process of the pre-trained diffusion model affects the overall structure and spatial layout, while the later stage focuses on more detailed visual effects. So content injection is performed in the early stage and style injection is performed in the later stage. A hyperparameter α is introduced, and the time separation nodes of different injections are calculated. T is the total number of steps in the diffusion model. The formula for split injection in the target branch is as follows:

[0060]

[0061] where t∈(0,T].

[0062] The method proposed in the present invention uses the pre-trained BLIP-2 encoder and CLIP text encoder, and the latent diffusion model Stable Diffusion as a diffusion model to achieve efficient and accurate image style conversion. In this invention, the input content image and style image are respectively processed through a specific process to generate a new image with the target style.

[0063] The present invention also provides an apparatus, comprising a memory and a processor, wherein: the memory is used to store a computer program that can be run on the processor; the processor is used to execute the steps of the above-mentioned diffusion model-based training-free image style transfer method when running the computer program.

[0064] The present invention also provides a storage medium storing a computer program, which, when executed by at least one processor, implements the steps of the above-mentioned diffusion model-based training-free image style transfer method.

[0065] In the present invention, once the corresponding text information and spatial information are extracted from the content image and the style image, a series of steps are taken to ensure the integrity of the content and the accuracy of the style during the image style transfer process. Specifically, the present invention relates to using the final result obtained by inverting the denoising diffusion implicit model DDIM in the content branch as the initial noise of the target branch, which is used as the starting point of the style transfer process. Such a processing method helps to maintain the basic features of the content image during the style transfer process, thereby improving the content integrity of the final generated image. Through the above steps, the method of the present invention can effectively balance the retention of content and the transfer of style during the image style transfer process, and generate images that are both faithful to the original content and have new style features.

[0066] Specifically, the method of the present invention first receives a content image, such as a picture of a bear, and a style image, such as a picture of a flame. In addition, an editable text input, such as "dog", is also received. In the processing flow, the content image and the style image are first denoised using the denoising diffusion implicit model DDIM inversion technology to generate a noisy image. The noisy image is then used as the input of the U-Net network for the subsequent style transfer process. It is worth noting that the input of the U-Net network is strictly provided by the noisy image obtained by the denoising diffusion implicit model DDIM inversion, rather than the output of the U-Net in the previous step, to ensure the accuracy of the information and the consistency of the process.

[0067] In the text input processing of the target branch, the method of the present invention feeds the style image into the BLIP-2 encoder for encoding to extract text information aligned with the style image, such as "flame". Subsequently, the extracted text information is combined with the specified text. In general, the specified text is empty in order to better learn the features of the style image. However, when it comes to image editing tasks, such as converting the structure of "bear" to the structure of "dog", the specified text is set to "dog" to guide the structural transformation of the image. Finally, the combined text information is used as text embedding in the target branch, thereby achieving precise control of the image content and style during the style transfer process.

[0068] In order to verify the effectiveness of the present invention, a comparative experiment was conducted with three other image style transfer models. These three network models are the currently popular network models InST (Inversion-based style transfer with diffusion models), DiffuseIT (Diffusion-based image translation using disentangled style and content representation) and CAP-VSTNet (CAP-VSTNet: content affinity preserved versatile style transfer.). All methods perform style transfer on content images and style images of the same size. Figure 4 The generation results shown in (a) to (d) are the generation results of InST, DiffuseIT, CAP-VSTNet and the algorithm proposed by the present invention. The present invention can make the output of the image style transfer model based on the diffusion model more harmonious with a small computational and time cost. It not only retains the content structure of the original content image, but also reflects the style of the style image, truly achieving a balance between content and style.

[0069] The above descriptions are merely embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in other related technical fields, should be included in the protection scope of the present invention.

Claims

1. A training-free image style transfer method based on a diffusion model, characterized in that: The steps include: (1) Generate text embedding corresponding to the style image based on the multimodal BLIP-2 encoder in BLIP-Diffusion; (2) Based on the residual module and self-attention module of U-Net, the key intermediate features are extracted from the content branch and the style branch through the denoising diffusion implicit model DDIM inversion technology; the intermediate features are used as the spatial features to be replaced in the target branch; (3) According to the different requirements of content and style, we selectively replace specific layers of the residual block and the self-attention module in content injection, and replace specific layers of the self-attention module in style injection to achieve the optimal fusion of content and style; (4) Utilizing the gradual characteristics of the diffusion model, the content injection and style injection in the target branch are performed separately.

2. The method for training-free image style transfer based on a diffusion model according to claim 1, characterized in that: The implementation process of step (1) is as follows: The pre-trained BLIP-2 encoder is denoted as F to convert the visual features of the style image into a representation aligned with the text; the text embedding of the CLIP text encoder ψ is retained in the diffusion model to enhance the expression of content information; the above two text embeddings are combined through a concatenation operation to guide the subsequent style transfer process: Among them, P c is an optional instructional text. s is the style image.

3. The method for training-free image style transfer based on a diffusion model according to claim 1, characterized in that: The process of extracting the intermediate features in step (2) is as follows: Each layer of the pre-trained diffusion U-Net consists of a residual block, a self-attention module for enhanced representation, and a cross-attention module for interaction with text conditions; for step t of the first layer in U-Net, with intermediate features The residual block will output The image enhanced based on the self-attention module is: in, is the final output of self-attention, is the mapping of the output of the residual block.

4. The method for training-free image style transfer based on a diffusion model according to claim 1, characterized in that: The different self-attention module replacement strategies in step (3) are: In content injection, the residual features in the target branch are injected Replaced with residual features in the content branch The query Q and key K in the self-attention module are also replaced in step t: in, It is a content branch Projected, The target branch projected In style injection, similar to content injection, in step t, the style space features are injected into the target branch, replacing the key K and value V elements: in, Style branch Projected, The target branch Projected.

5. The method for training-free image style transfer based on diffusion model according to claim 1, characterized in that: In step (3), the specific layers of the residual block and the self-attention module are selectively replaced during content injection, namely, the 3rd to 8th layers in the residual block and the 4th to 11th layers in the self-attention module are replaced.

6. The method for training-free image style transfer based on a diffusion model according to claim 1, characterized in that: In step (3), the specific layer of the self-attention module that is replaced in the style injection is the 4th to 11th layers in the self-attention module.

7. The method for training-free image style transfer based on a diffusion model according to claim 1, characterized in that: Step (4) implements the step-by-step injection in the target branch as follows: Content injection is performed in the early stage of the reverse process of the pre-trained diffusion model, and style injection is performed in the later stage; and a hyperparameter α is introduced to calculate the time separation nodes of different injections The formula for split injection in the target branch is as follows: Where t∈(0,T]; T is the total number of steps in the diffusion model.

8. The method for training-free image style transfer based on a diffusion model according to claim 7, characterized in that: The value of the hyperparameter α is 0.

2.

9. A device, characterized in that: comprising a memory and a processor, wherein: A memory for storing computer programs that can be run on the processor; A processor, configured to execute the steps of the training-free image style transfer method based on a diffusion model as described in any one of claims 1 to 8 when running the computer program.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by at least one processor, the steps of the training-free image style transfer method based on a diffusion model as described in any one of claims 1 to 8 are implemented.

Images