A Flexible and Consistent Appearance Transfer Method Based on a Training-Free Diffusion Model

Abstract

This invention proposes a flexible and consistent appearance transfer method based on a training-free diffusion model. Without additional training, this method achieves high-quality appearance transfer while maintaining consistency between the source image structure and the background. Furthermore, it supports more flexible control, including appearance transfer of specified regions and transfer of multiple appearances. Specifically, the method employs dual guided branches to extract the appearance features of the reference image and the structural features of the source image, respectively, and then precisely fuses them through an improved attention mechanism (Mask-Appearance-Attention) to ensure consistency between the structure and the background.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to image editing and personalized generation techniques. More specifically, this invention relates to a flexible and consistent appearance transfer method based on a training-free diffusion model, aiming to achieve high-quality appearance transfer while ensuring the consistency of the source image structure and background, and providing users with more flexible control to achieve the transfer of specified regions and multiple appearances. Background Technology

[0002] In recent years, generative artificial intelligence has made groundbreaking progress and has also spawned many more interesting and personalized tasks. Among them, appearance transfer is a common problem, which aims to seamlessly integrate the appearance of a reference image into a source image.

[0003] Early methods for achieving appearance transfer mainly consisted of Generative Adversarial Networks (GANs) and autoencoders. These methods can achieve simple appearance transfer, but require a lot of computational resources for training, and often exhibit inconsistency and instability when dealing with complex image structures and backgrounds.

[0004] In recent years, with the emergence of diffusion models, image editing methods based on diffusion models have shown significant advantages in various image generation and editing tasks, and some studies have also attempted to apply diffusion models to appearance transfer tasks.

[0005] For example, DiffuseIT uses a pre-trained DINO-ViT model to extract structural and appearance features of images, thereby guiding the denoising process to achieve appearance transfer. Due to the powerful generative capabilities of the diffusion model, DiffuseIT achieves good results, but it still faces problems such as poor structural consistency and inaccurate background transfer. DiffEditor proposes another image editing method based on the diffusion model, integrating image and text embeddings to guide the denoising process, thus improving the accuracy and flexibility of transfer. DiffEditor can guarantee background consistency to a certain extent, but it still performs poorly in terms of structural consistency, especially when dealing with scenes with complex structures or backgrounds, where its effectiveness is limited. Furthermore, although this method enhances the flexibility of appearance transfer, its performance remains unstable for complex image content, easily resulting in background interference or loss of detail. Cross-image aligns the semantics of the reference and target images through the self-attention mechanism of the diffusion model, enabling the generated image to query the features of the reference image, thereby achieving appearance transfer. While this method can achieve high-quality appearance transfer, it has significant shortcomings in ensuring structural consistency, especially in the processing of complex backgrounds, where the background of the reference image may unnecessarily migrate into the target image, leading to a chaotic generated image. Furthermore, most existing methods lack flexible regional control and cannot perform appearance migration for specific regions.

[0006] In summary, although these methods have made some progress in appearance transfer tasks, they still face many challenges. In particular, existing methods generally have limitations in ensuring the consistency of the source image structure and background, and in achieving flexible and diverse appearance transfers. Most methods fail to distinguish the boundaries of the appearance transfer region, leading to the leakage of background information and the loss of structural features. Furthermore, existing methods struggle to provide flexible transfer control, failing to meet the increasingly diverse user needs. Therefore, ensuring the consistency of the source image structure and background during the transfer process, while providing users with flexible control, remains a key challenge. Summary of the Invention

[0007] To address the above problems, this invention proposes a flexible and consistent appearance transfer method based on a training-free diffusion model, comprising the following steps:

[0008] S1, based on the source image and the reference image, uses a pre-trained diffusion model to transfer the appearance of the reference image to the source image while keeping the structure and background of the source image unchanged;

[0009] S2, segment the source image and reference image, and add noise to the source image and reference image through DDIM inversion to prepare for subsequent input;

[0010] S3, construct a dual-guided branch, each branch is a separate diffusion model. By analyzing the influence of the self-attention mechanism on structure and appearance, extract the structural features of the source image and the appearance features of the reference image respectively, in preparation for subsequent fusion;

[0011] S4, construct the generation branch, which is used to generate the final image and also provides the structural features of the source image;

[0012] S5. An improved attention mechanism, Mask-Appearance-Attention (MAA), is designed to fuse the structural features of the source image and the appearance features of the reference image. An appropriate time step and U-Net layer are selected, and the self-attention in the generation branch is replaced with MAA. Specifically, the following steps are included:

[0013] S51, The structural and appearance features obtained by fusing guided branches are fused and guided by combining attention from different branches, allowing the source image to... and Separate and reference images The computation of attention can be represented as:

[0014] (1)

[0015] (2)

[0016] in, and The query parameters represent the two branches of the source image. This represents the attention parameters of the reference image. and This represents the calculation results of two attention points;

[0017] S52, restricting the appearance transfer area to ensure the consistency of the source image structure and background, limits the attention operation area through a mask, ensuring that combined attention only performs in the specified appearance transfer area, preventing background information leakage. This part can be represented as:

[0018] (3)

[0019] (4)

[0020] in These represent the masks for the corresponding main regions. and This represents the attention calculation result of the two combined masks;

[0021] S53, Integrate the two attention results. Depend on Calculations show that The source image reconstruction process from the guiding branch, Contains more accurate structural information. By generating branches The obtained information contains richer appearance information. Linear integration of the two allows for a better fusion of structural and appearance features. This part can be represented as:

[0022] (5)

[0023] in This represents the appearance coefficient, which can be used to control the intensity of appearance migration;

[0024] S54, Calculate the attention for the background region, ensuring background consistency. The background region undergoes regular self-attention, constrained by a background mask. The final attention result consists of the transfer region and the background region, which can be represented as:

[0025] (6)

[0026] (7)

[0027] in For the final attention calculation result, express , It will replace the self-attention result of the generated branch and perform subsequent calculations;

[0028] S55, Select an appropriate time step and U-Net layer to replace MAA, and set the time step and layer thresholds. If and only if time step Greater than and number of layers Greater than When the self-attention of the generated branch is replaced with MAA, this part can be represented as:

[0029] (8)

[0030] The generated branch denoises over multiple time steps to produce a satisfactory appearance transfer result.

[0031] S6 allows for flexible appearance migration, enabling users to customize masks to migrate appearances in specified areas and migrate multiple appearances.

[0032] S1 specifically includes the following steps:

[0033] S11, Prepare source image and reference image The source image and the reference image each have at least one subject, which serves as the target of migration and the source of appearance. The subject can be an animal, a building, a vehicle, or food.

[0034] S2 specifically includes the following steps:

[0035] S21, use the Segment Anything Model (SAM) to segment the source image and reference image to obtain the segmentation map of the corresponding subject. ;

[0036] S22, DDIM inversion is performed on the source and reference images to obtain the corresponding noise map. As input to subsequent models, this part can be described as follows:

[0037] (9)

[0038] in In order to be in Time step prediction That is, predicting the latent space of the input image. Is Noise in time step prediction It is a noise scaling factor that is related to the time step.

[0039] The specific steps involved in S3, which constructs a dual-guided branch and extracts the structural features of the source image and the appearance features of the reference image, are as follows:

[0040] S31, the input to the guiding branch is the noise map of the source image and the reference image. It comprises two simultaneously and independently performing diffusion models to reconstruct the source image and the reference image, respectively. The method of this invention mainly focuses on the self-attention layer of the diffusion model U-Net. During the reconstruction process, conventional self-attention calculation is first performed, which can be described as follows:

[0041] (10)

[0042] (11)

[0043] (12)

[0044] in Representing different parameter matrices, This represents the three parameters of the self-attention layer in a dual-guided branch. These represent the feature maps of the latent space of the input image in different branches within different U-Net layers. express The dimension of the feature;

[0045] S32 extracts the structural features of the source image and the appearance features of the reference image from a conventional self-attention layer, and analyzes the self-attention mechanism. The impact on structure and appearance during reconstruction is considered... It has a greater impact on the structure of the image, while This refers to the appearance of the image, therefore the reconstruction branch retains the source image. and the reference image reconstruction branch Used for subsequent feature fusion.

[0046] The specific steps involved in building and generating branches using S4 are as follows:

[0047] S41, the noise map of the source image is used as the input to the generation branch, denoted as... The generated branch is also an independent diffusion model, and its self-attention mechanism can be calculated as follows:

[0048] (13)

[0049] in, This represents the parameters of the self-attention layer in the generation branch. To generate feature maps for different layers of the branch U-Net;

[0050] S42, extract features from the attention of the generated branch. The input to the generated branch is the noise map of the source image, retaining... To obtain richer structural features.

[0051] The S6 method for achieving flexible appearance migration specifically includes the following steps:

[0052] S61 enables appearance migration of a specified area. Users can customize a mask to specify the migration area of ​​interest. This mask can be part of the main body of the source image.

[0053] S62 enables the transfer of various appearances, given multiple reference images and corresponding transfer regions, which can be different parts of the main body of the source image.

[0054] Beneficial Effects: Compared with existing technologies, this invention provides a flexible and consistent appearance transfer method based on a training-free diffusion model, which can achieve more consistent and flexible appearance transfer and produces the following beneficial effects:

[0055] 1. Efficient appearance transfer: Most existing methods require a lot of computation for training, while this invention guides the pre-trained diffusion model by designing attention, which can obtain high-quality transfer results in just 20 seconds without additional training or fine-tuning, significantly improving the generation efficiency.

[0056] 2. More Consistent Appearance Transfer: Existing methods often operate on the entire image as a unit, failing to distinguish the boundary between the transfer region and the background. This leads to problems such as loss of source image structure and mis-transfer of the reference image background. This invention employs a dual-guided branching approach to extract structural and appearance features and designs an improved attention mechanism for fusion, operating separately on the transfer region and background region to ensure consistency between structure and background.

[0057] 3. More Flexible Control: This invention offers more flexible control, allowing users to customize masks to specify regions of interest and migrate only the appearance of the reference image to the corresponding region. Furthermore, given multiple reference images and corresponding regions, this invention can migrate the appearance of each image to the source image, resulting in interesting generated results. This more flexible control also meets the increasingly diverse needs of users.

[0058] 4. Broad application prospects: This invention is applicable to various image generation and editing tasks, such as artistic creation, advertising production, and virtual try-on, providing users with a more realistic and convenient experience, reducing costs for creators, and has high commercial value. Attached Figure Description

[0059] Figure 1 The flowchart shows a flexible and consistent appearance transfer method based on a training-free diffusion model, detailing each step from the source image and reference image to the final generated image.

[0060] Figure 2 The comprehensive demonstration results of this invention include global (main) appearance migration, migration of specified areas, and migration of multiple appearances.

[0061] Figure 3 This describes the visual effect of the method of the present invention on the appearance migration in a specified area.

[0062] Figure 4 The method of this invention achieves various appearance migration results.

[0063] Figure 5 This is a visual comparison diagram of the method of the present invention and other methods.

[0064] Figure 6 , Figure 7 and Figure 8 This is a comparison diagram of the present invention in terms of structural consistency, appearance similarity, and background consistency. Detailed Implementation

[0065] The flexible and consistent appearance transfer method based on a training-free diffusion model of the present invention achieves high-quality appearance transfer by employing dual guided branches and an improved attention mechanism (MAA). It ensures the consistency of the source image structure and background and provides flexible control, including appearance transfer of specified regions and transfer of multiple appearances. The specific implementation process is as follows.

[0066] The invention will be further described below with reference to specific embodiments.

[0067] Example 1:

[0068] like Figure 1 As shown, given a source image and a reference image, a mask image is first obtained through SAM, and a noise map is obtained through DDIMInversion, both of which are input into the model. The model includes a guiding branch and a generating branch, which are performed simultaneously. The guiding branch reconstructs the source and reference images, preserving structure and appearance-related parameters. The generating branch also preserves structure-related parameters. The preserved parameters are input into the improved attention mechanism MAA for fusion, and a mask is used to limit the attention computation area to ensure the consistency between the structure and the background. The attention calculation result obtained after fusion returns to the generating branch for subsequent operations. After several time steps, the generating branch obtains the final result.

[0069] Figure 2 The comprehensive demonstration results of this invention include global (main) appearance migration, migration of specified areas, and migration of multiple appearances.

[0070] Figure 3 This describes the visual effect of the method of the present invention on the appearance migration in a specified area.

[0071] Figure 4 The method of this invention achieves various appearance migration results.

[0072] Figure 5 This is a visual comparison of the method of this invention and other methods. It achieves high-quality appearance transfer while simultaneously ensuring consistency between the structure and the background.

[0073] Figure 6 , Figure 7 and Figure 8 The figures show a comparison of the present invention in terms of structural consistency, appearance similarity, and background consistency. It can be seen that the method presented in this paper is superior to existing methods in terms of structural and background consistency, and is also highly competitive in terms of appearance similarity.

[0074] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0075] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A flexible and consistent appearance transfer method based on a training-free diffusion model, characterized in that, Includes the following steps: S1, based on the source image and the reference image, uses a pre-trained diffusion model to transfer the appearance of the reference image to the source image while keeping the structure and background of the source image unchanged; S2, segment the source image and reference image, and add noise to the source image and reference image through DDIM inversion to prepare for subsequent input; S3, construct a dual-guided branch, each branch is a separate diffusion model. By analyzing the influence of the self-attention mechanism on structure and appearance, extract the structural features of the source image and the appearance features of the reference image respectively, in preparation for subsequent fusion; S4, construct the generation branch, which is used to generate the final image and also provides the structural features of the source image; S5. An improved attention mechanism, Mask-Appearance-Attention (MAA), is designed to fuse the structural features of the source image and the appearance features of the reference image. An appropriate time step and U-Net layer are selected, and the self-attention in the generation branch is replaced with MAA. Specifically, the following steps are included: S51, The structural and appearance features obtained by fusing guided branches are fused and guided by combining attention from different branches, allowing the source image to... and Separate and reference images The computation of attention can be represented as: (1) (2) in, and The query parameters represent the two branches of the source image. This represents the attention parameters of the reference image. and This represents the calculation results of two attention points; S52, restricting the appearance transfer area to ensure the consistency of the source image structure and background, limits the attention operation area through a mask, ensuring that combined attention only performs in the specified appearance transfer area, preventing background information leakage. This part can be represented as: (3) (4) in These represent the masks for the corresponding main regions. and This represents the attention calculation result of the two combined masks; S53, Integrate the two attention results. Depend on Calculations show that The source image reconstruction process from the guiding branch. Contains more accurate structural information. By generating branches The obtained information contains richer appearance information. Linear integration of the two allows for a better fusion of structural and appearance features. This part can be represented as: (5) in This represents the appearance coefficient, which can be used to control the intensity of appearance migration; S54, Calculate the attention for the background region, ensuring background consistency. The background region undergoes regular self-attention, constrained by a background mask. The final attention result consists of the transfer region and the background region, which can be represented as: (6) (7) in For the final attention calculation result, express , It will replace the self-attention result of the generated branch and perform subsequent calculations; S55, Select an appropriate time step and U-Net layer to replace MAA, and set the time step and layer thresholds. If and only if time step Greater than and number of layers Greater than When the self-attention of the generated branch is replaced with MAA, this part can be represented as: (8) The generated branch denoises at multiple time steps to generate a satisfactory appearance transfer result; S6 allows for flexible appearance migration, enabling users to customize masks to migrate appearances in specified areas and migrate multiple appearances.

2. The flexible and consistent appearance transfer method based on a training-free diffusion model as described in claim 1, characterized in that, S1 specifically includes the following steps: S11, Prepare source image and reference image The source image and the reference image each have at least one subject, which serves as the target of migration and the source of appearance. The subject can be an animal, a building, a vehicle, or food.

3. The flexible and consistent appearance transfer method based on a training-free diffusion model as described in claim 1, characterized in that, S2 specifically includes the following steps: S21, use the Segment Anything Model (SAM) to segment the source image and the reference image to obtain the segmentation map of the corresponding subject. ; S22, DDIM inversion is performed on the source and reference images to obtain the corresponding noise map. As input to subsequent models, this part can be described as follows: (9) in Indicates at time step Predicted latent space representation of the input image , Is Noise in time step prediction It is a noise scaling factor that is related to the time step.

4. The flexible and consistent appearance transfer method based on a training-free diffusion model as described in claim 1, characterized in that, The specific steps involved in S3, which constructs a dual-guided branch and extracts the structural features of the source image and the appearance features of the reference image, are as follows: S31, the input to the guiding branch is the noise map of the source image and the reference image. It comprises two simultaneous and independent diffusion models, which reconstruct the source image and the reference image respectively. In the self-attention layer of U-Net, during the reconstruction process, conventional self-attention calculation is performed first. This part can be described as follows: (10) (11) (12) in Representing different parameter matrices, This represents the three parameters of the self-attention layer in a dual-guided branch. These represent the feature maps of the latent space of the input image in different branches within different U-Net layers. express The dimension of the feature; S32 extracts structural features of the source image and appearance features of the reference image from a conventional self-attention layer, demonstrating the self-attention mechanism. The impact on structure and appearance during reconstruction is considered... It has a greater impact on the structure of the image, while This refers to the appearance of the image, therefore the reconstruction branch retains the source image. and the reference image reconstruction branch Used for subsequent feature fusion.

5. The flexible and consistent appearance transfer method based on a training-free diffusion model as described in claim 1, characterized in that, The S4 branch generation process specifically includes the following steps: S41, the noise map of the source image is used as the input to the generation branch, denoted as... The generated branches are also independent diffusion models, and their self-attention mechanism can be calculated as follows: (13) in, This represents the parameters of the self-attention layer in the generation branch. To generate feature maps for different layers of the branch U-Net; S42, extract features from the attention of the generated branch. The input to the generated branch is the noise map of the source image, retaining... To obtain richer structural features.

6. A flexible and consistent appearance based on a training-free diffusion model as described in claim 1. The migration method is characterized by, The S6 method for achieving flexible appearance migration specifically includes the following steps: S61 enables appearance migration of a specified area. Users can customize a mask to specify the migration area of ​​interest. This mask can be part of the main body of the source image. S62 enables the transfer of various appearances, given multiple reference images and corresponding transfer regions, which can be different parts of the main body of the source image.

Images