A training-free image harmonization completion method

Abstract

This invention proposes a training-free image harmonization and completion method, relating to the field of computer vision technology. The method involves acquiring a masked input image, a corresponding completion mask, and text prompts; inputting the input image, completion mask, and text prompts into a pre-trained stable diffusion model; dividing the self-attention map based on the completion mask into interactions within the masked region, interactions within the unmasked region, and interactions between the masked and unmasked regions; selectively weighting the interactions between the masked and unmasked regions to generate a reweighted self-attention map; adjusting the keys and values ​​in the self-attention mechanism to introduce style features from the unmasked region into the masked region, ensuring style consistency between the masked and unmasked regions; and completing the image completion based on the reweighted self-attention map and style harmony optimization, outputting a harmonized completed image.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and specifically to a training-free image harmonic completion method. Background Technology

[0002] With the development of digital image processing technology, image completion, as an important branch of digital image processing, has been widely used in digital art, design, personalized advertising, and other fields. Its core task is to fill in missing areas marked by masks in an image, ensuring that the completed content maintains structural and stylistic consistency with the original image. Current image completion technology has evolved from traditional texture filling to deep learning model-based techniques, with each stage exhibiting different characteristics and limitations.

[0003] Traditional texture filling techniques fill missing areas by searching for similar texture blocks in an image. These techniques can only generate general textures and cannot generate customized content based on specific needs such as text descriptions. Furthermore, they are poor at completing complex structural areas and cannot meet the demands for high-precision rendering.

[0004] While generative adversarial networks (GANs)-based completion techniques have improved the diversity of generated content to some extent, they are prone to pattern collapse during training, resulting in monotonous generated content. Furthermore, they are difficult to balance structural integrity and stylistic consistency during the completion process, which limits their application in high-quality completion scenarios.

[0005] Image completion techniques based on diffusion models have become mainstream in the field of image completion in recent years. Their core mechanism is based on a progressive noise addition-denoising Markov chain process: during training, the model progressively adds Gaussian noise to the original image until a random noise map is formed, recording the noise features at each step; during inference, the model learns to reverse the noise process through a noise predictor, gradually recovering a clear image from the random noise. Compared with traditional texture filling and the first two generations of GAN-based completion techniques, this technique has significant advantages. On the one hand, by combining multimodal information such as text prompts encoded by the CLIP model, it achieves customized text-guided completion. For example, users can specify the object type and style features of the completion area through text commands, and the model generates content that meets expectations based on the commands, greatly improving the flexibility and controllability of completion. On the other hand, thanks to the latent diffusion model architecture, it has a natural advantage in generating high-quality image details, further expanding the application space of high-resolution image completion.

[0006] While existing technologies have made progress in the field of image completion, they still have the following core shortcomings:

[0007] First, existing image completion techniques rely on extensive training and have poor versatility. Fine-tuning methods are highly dependent on training data of a specific style. If the application scenario is switched to an untrained style, the harmony between the completed region and the background style will significantly decrease. At the same time, fine-tuning of these methods requires a large amount of computational resources, making it difficult for them to quickly adapt to new scenarios and limiting their versatility.

[0008] Secondly, the generated images may suffer from insufficient structural realism and disordered spatial layout. The self-attention layers of methods such as SDI and CNI have limitations and cannot effectively distinguish between masked and unmasked regions. Since the principal component features of the two are similar, background features interfere with the completion process, resulting in structural defects or spatial misalignment in the generated content, which seriously affects the structural rationality of the completion.

[0009] Finally, there is poor style coordination and a prominent problem of visual disjointness. Methods such as BLD and CNI have difficulty effectively transferring the style of unmasked areas to masked areas. Even if the SDI method can generate content that matches the text description, the style and background still cannot be consistent, resulting in a significant style disconnect. Summary of the Invention

[0010] To address the aforementioned technical problems, this invention proposes a training-free image harmonization and completion method, comprising the following steps:

[0011] S1: Obtain the masked input image, the corresponding completion mask, and the text prompt;

[0012] S2: Input the input image, the completion mask, and the text prompts into the pre-trained stable diffusion model;

[0013] S3: Based on the completion mask, the self-attention map is divided into interactions within the masked region, interactions within the unmasked region, and interactions between the masked and unmasked regions. The interactions between the masked and unmasked regions are selectively weighted and adjusted to generate a reweighted self-attention map.

[0014] S4: Adjust the keys and values ​​in the self-attention mechanism to introduce style features of the unmasked region into the masked region, ensuring style consistency between the masked and unmasked regions;

[0015] S5: Based on the reweighted self-attention map and style coordination optimization, complete the image completion and output a harmonized completed image.

[0016] Further, in step S3, the completion mask is adjusted to the same dimension as the self-attention map and flattened into a one-dimensional vector; the interaction within the masked region and the interaction within the unmasked region are calculated using the one-dimensional vector, and the interaction between the two types of regions is readjusted; a soft masking mechanism is used to smooth the mask, wherein the smoothing factor τ is used to control the smoothing intensity of the mask.

[0017] Further, in step S4, the mean of the keys and values ​​of the unmasked regions is extracted; the original keys and values ​​of the masked regions are replaced with the mean; and a new self-attention map and latent features are calculated based on the updated keys and values, wherein the hyperparameter λ is used to control the intensity of style transfer.

[0018] Furthermore, it also includes step S6: applying attention-guided loss, which extracts a cross-attention map during the denoising process and calculates the loss of alignment with the text prompt token within the masked region to enhance the consistency between the completed content and the text prompt.

[0019] Furthermore, the calculation of the attention guidance loss includes: extracting cross-attention maps with resolutions of 16 and 32 at each time step; minimizing the difference between the target attention and the predicted attention within the mask using a loss function, so that the attention of the text prompt token is focused on the mask region.

[0020] Furthermore, the denoising process of the pre-trained stable diffusion model is divided into two stages: the first stage, from time step 0 to ηT, focuses on improving the structural realism of the completed content; the second stage, from time step ηT to T, focuses on enhancing the style coordination between the masked and unmasked regions; where η is a hyperparameter used to control the stage division.

[0021] Furthermore, the pre-trained stable diffusion model is an image generation model that does not require additional training, and all optimization steps are applied during the inference process.

[0022] Furthermore, the self-attention masking strategy and style reconciliation optimization are implemented in the encoder and decoder of the U-Net architecture, respectively, to optimize feature propagation and style transfer.

[0023] To address the shortcomings of existing technologies, this invention aims to solve the following core technical problems:

[0024] How to achieve cross-style image completion without additional training or fine-tuning, thereby improving the method's versatility and adaptability: Existing methods typically rely on extensive training or fine-tuning, resulting in poor versatility. This invention achieves cross-style image completion without additional training or fine-tuning, thus improving the method's adaptability and versatility, enabling rapid adaptation to different styles and application scenarios. How to optimize the self-attention mechanism of the diffusion model to effectively distinguish between masked and unmasked regions, avoid background feature interference, and ensure the structural authenticity of the completed content (e.g., complete shape and reasonable spatial layout): Current diffusion models often suffer from background features interfering with the generation of masked regions when processing completion tasks. By optimizing the self-attention mechanism, this invention can effectively distinguish between masked and unmasked regions, reduce background interference, ensure the structural authenticity of the completed content, and guarantee the generated shape is complete and has a reasonable spatial layout. How to utilize the inherent characteristics of the diffusion model to establish a style transfer mechanism from the background to the completed region, and achieve style coordination between the completed region and the background: Existing methods have shortcomings in style coordination. This invention utilizes the inherent characteristics of the diffusion model to design a style transfer mechanism from the background to the completed region, ensuring that the generated completed region is style-coordinated with the background, avoiding style disconnect, and improving the overall visual consistency of the image. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0026] Figure 1 This is a general framework diagram of the training-free image harmonization and completion method provided by the present invention.

[0027] Figure 2 This is a schematic diagram showing the qualitative comparison results of the present invention with existing technologies on the Stylized-COCO dataset.

[0028] Figure 3 This is a schematic diagram showing the qualitative comparison results of the present invention with existing technologies on the Stylized-OpenImages dataset.

[0029] Figure 4 This is a schematic diagram showing the qualitative comparison results of ablation experiments of different components in this invention.

[0030] Figure 5 This is a schematic diagram showing the comparative experimental results of different self-attention masking strategies in this invention.

[0031] Figure 6 This is the hyperparameter of style transfer intensity in this invention. A visual diagram illustrating the impact on the generated results.

[0032] Figure 7 It is the smoothing factor in this invention. A visual diagram illustrating the impact on the generated results.

[0033] Figure 8 This is a schematic diagram of the image completion result of the method of the present invention in a general natural scene. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0035] The overall framework of this invention is based on the U-Net architecture of a stable diffusion model, aiming to solve image inpainting tasks. The model input consists of a masked original image, a completion mask, and text prompts. The mask marks the areas to be inpainted, while the text prompts provide content guidance for the inpainting process. Next, the image undergoes preliminary processing using the basic diffusion model to generate a partially inpainted image. To enhance the structural fidelity of the inpainted results, the model introduces a self-attention masking strategy and attention-guided loss to further improve inpainting accuracy. Subsequently, to ensure stylistic consistency in the inpainted results, the model adjusts... , , A masking strategy for key values ​​is employed to optimize the harmony of styles both inside and outside the mask. Ultimately, after these processing steps, the model outputs a complete restored image, achieving efficient and high-quality output in the restoration task.

[0036] Example 1

[0037] The overall framework of the present invention is as follows Figure 1 As shown. The self-attention map in the diffusion model can effectively capture the overall layout of the image. This invention further verifies the role of self-attention blocks in image completion through experiments, a point often overlooked in existing methods. Given an incomplete image, a completion mask (mask region value is 1), and text prompts, this invention uses a stable diffusion model for image completion, with results as shown. Figure 1 As shown, it can be observed that the rabbits generated by stable diffusion have incomplete bodies and do not match the geometry of their surroundings.

[0038] To analyze the root causes of the problem, this invention extracts self-attention maps from the U-Net network and performs principal component analysis. Since skip connections inevitably couple the representations of the encoder and decoder, leading to difficulties in feature unwrapping, this invention focuses on analyzing the source of information propagation in the U-Net encoder. This invention finds that the stable diffusion completion approach, which classifies masked and unmasked regions as the same principal component in the U-Net encoder, is not ideal. Existing research indicates that the U-Net encoder extracts features from noisy latent input, and the decoder uses these features to predict the image's spatial layout. When the encoder treats masked and unmasked regions as similar components, it struggles to capture the spatial relationship between the completed content and the surrounding areas, resulting in an inaccurate layout.

[0039] To address this issue, this invention proposes a self-attention masking strategy. The self-attention mechanism effectively captures the relationships between image patches. Based on this, this invention divides the self-attention map into three regions according to the completion task: interactions within the masked region (object-object, denoted as "obj-obj"), interactions within the unmasked region (background-background, denoted as "bg-bg"), and interactions between the masked and unmasked regions (object-background, denoted as "obj-bg"). By selectively masking the 'obj-bg' interactions, the method of this invention prevents background information from penetrating the object region, improving the targeting and accuracy of the completion process. This strategy reduces background interference and improves the accuracy and quality of the completed content.

[0040] First, this invention will use the given completion mask. Adjusted to match the self-attention map in each self-attention block. With consistent dimensions, a one-dimensional complete mask is obtained. H and W are the height and width of the current feature map, which are flattened into a one-dimensional vector and incorporated into it, as shown in the following formula:

[0041] ;

[0042] ;

[0043] ;

[0044] in, This indicates "obj-obj" interaction. This indicates a "bg-bg" interaction, corresponding to... Figure 2 The pink and blue areas in the middle.

[0045] During the encoding phase, this invention selectively handles the "obj-bg" interaction, corresponding to... Figure 1 The gray areas in the image are masked and then processed using the improved method. Replace the original self-attention map This adjustment makes the potential features of the masked region more focused on the completed region, reducing reliance on the unmasked region. For example... Figure 2 As shown, the masked and unmasked regions have different principal components. Unlike previous methods that only distinguish between self-attention components inside and outside the mask, this invention performs a more refined division of the self-attention map to meet the needs of the completion task. This optimization enables stable diffusion completion to better distinguish between masked and unmasked regions, improving structural realism.

[0046] However, directly introducing a one-dimensional completion mask into a self-attention block may excessively hinder information exchange between masked and unmasked regions. Therefore, in practical applications, this invention employs a soft masking strategy to obtain a soft completion mask. .

[0047] ;

[0048] in, This is a smoothing factor.

[0049] The structural realism of image completion depends not only on the relationship between the object and the background, but also on the alignment between the completed object and the text cue. To better apply this concept to completion tasks, this invention proposes an attention-guided loss method—this method focuses on tokens associated with the cue within the masked region. The specific steps are as follows:

[0050] At each time step, the invention performs a denoising process and extracts cross-attention maps at resolutions of 16 and 32 (these two resolutions capture the richest semantic information). Subsequently, the average cross-attention map is calculated at each resolution. , where L represents the number of tokens in the text prompt. For the i-th token, Incorporating its attention map, the formula is as follows:

[0051] ;

[0052] in, This represents the attention weight of the i-th token in the average cross-attention map to the j-th image patch containing the noisy latent variable. This represents the attention weight of the i-th token in the cross-attention graph corresponding to the soft-painted mask region on the j-th image patch of the noisy latent variable.

[0053] Next, enumerate the attention graphs. And define the following loss target :

[0054] ;

[0055] By setting the attention to the unmasked region to zero, and then minimizing the difference loss between the target attention and the predicted attention within the mask, the token attention can be concentrated as much as possible in the masked region.

[0056] This invention treats style harmony enhancement as a local style transfer task. Traditional style transfer methods typically extract attributes such as color and texture from a reference image and apply them to the target content image. In the completion task presented in this paper, this mechanism is modified: unmasked regions serve as style references, and masked regions serve as content references. Unlike traditional style transfer across images, this invention achieves local style transfer within a single image, requiring the extraction of style information from specific regions while maintaining consistency in style and content across the entire image.

[0057] The key to the self-attention module of the U-Net decoder Sum This can be used to extract style information. Based on this, the present invention proposes a mask adjustment key-value strategy, which introduces the extracted style features into the mask area to achieve style consistency. For example, this invention utilizes soft-filling masks. right Perform an update to obtain the updated K value. :

[0058] ;

[0059] in, This represents the mask value corresponding to the i-th block in the soft completion mask (1 indicates that it is within the completion area, and 0 indicates that it is outside the completion area). Indicates the image number 1 The characteristics of each block represent This represents the mean value of the features of the unmasked region. The present invention believes that... The overall style features of the incomplete image are represented, and therefore, these features are used to replace the original style features of the masked region to ensure that the style of the masked region remains consistent with that of the unmasked region. This invention updates V in the same way to obtain the updated V value. :

[0060] ;

[0061] And based on this, a new self-attention map is calculated. and potential characteristics :

[0062] ;

[0063] ;

[0064] in, is a hyperparameter used to control the intensity of style transfer, and d is a scaling factor with a size of K dimensions.

[0065] The diffusion model first generates a rough outline and shape of the object, and then gradually optimizes the details during the denoising process. Utilizing this characteristic, this invention divides the denoising process into two stages and introduces hyperparameters. Total time steps Divided into two stages: and .exist In this phase, the focus is on improving the structural authenticity of the supplementary content; In this phase, the focus is on enhancing the stylistic consistency between masked and unmasked regions. This design significantly reduces the computational cost during the inference process.

[0066] Example 2

[0067] To further verify the technical effectiveness of this invention, a series of experiments were designed for analysis. In the experiments, this invention was compared with several advanced text-guided image completion methods.

[0068] I. Construction of Experimental Dataset

[0069] Existing commonly used image completion datasets are mainly geared towards natural image scenes and cannot meet the needs of this invention to test the coordinated completion effect in multi-style scenes. Therefore, this invention constructs two types of stylized datasets:

[0070] The Stylized-COCO dataset utilizes style transfer techniques to transform images from the MSCOCO dataset into different artistic styles. This dataset includes two types of annotations: segmentation masks and bounding box masks. Text conditions use the original image description information from the MSCOCO dataset.

[0071] Stylized-OpenImages dataset: Following the same construction process as Stylized-COCO, this dataset is formed by using the OpenImages dataset as the base image source, completing stylization transformation and mask annotation.

[0072] II. Selection of Experimental Evaluation Indicators

[0073] Traditional evaluation metrics used in image completion are insufficient to adequately measure the diversity and style consistency of completion results in the scenario described in this invention. Therefore, this experiment employs the following four metrics to comprehensively evaluate the completion performance of each model. The specific evaluation metrics and their core functions are as follows:

[0074] CLIP Score (CS): Used to quantify the semantic alignment between the completed content and the given text prompt. A higher score indicates a stronger match between the completed result and the text description.

[0075] Image reward (IR): Based on a model trained on human preferences, it was initially applied to text-to-image generation tasks. This invention uses it to evaluate the structural realism of completed images, i.e., the consistency between the completed region and the unmasked region in terms of spatial structure and object shape.

[0076] Aesthetic Score (AS): Calculated by a linear model trained on a real image quality scoring dataset, it is used to quantify the compatibility between the completed image and the original image in dimensions such as color tone, texture style, and artistic quality.

[0077] CLIP Maximum Mean Difference (CMMD): This evaluates the difference in feature distribution between the reference image set (original image) and the generated image set (completed image) in the CLIP embedding space by calculating the MMD (maximum mean difference) distance between them. The smaller the distance, the closer the overall distribution of the completed image is to the original image, and the more stable the completion effect.

[0078] III. Comparison with state-of-the-art methods

[0079] 1. Qualitative assessment

[0080] The test results for the completion task are shown below. Figure 2 and Figure 3 On the Stylized-COCO dataset, this invention observes that BLD and CNI typically generate spatially inconsistent results. For example, Figure 2 In the first line, although the completed content matches the text description, the cat's shape and outline are inconsistent with the surrounding environment. This inconsistency stems from the fact that these methods fail to fully utilize the self-attention mechanism during denoising—in a self-attention map, masked and unmasked regions are treated as the same principal component, thus limiting the diffusion model's ability to model spatial relationships. In contrast, this invention introduces a self-attention masking strategy into the U-Net encoder, effectively overcoming this limitation and significantly improving structural realism.

[0081] The comparative method has made significant progress in terms of style harmony. For example, Figure 2 In the second row, BLD and CNI generate realistic horses in the masked region, but the style of the completed region does not match the unmasked region, resulting in visual inconsistency. In contrast, this invention maintains stylistic consistency between the masked and unmasked regions while aligning the cue words. This achievement is attributed to the mask adjustment key-value strategy proposed in this invention, which extracts style information from the unmasked region and integrates it into the masked region. On the Stylized-OpenImages dataset, this invention also achieves consistent completion results, validating its effectiveness across different styles and scenes.

[0082] Table 1 presents a quantitative comparison between the method of this invention and the state-of-the-art methods on the Stylized-COCO dataset; Table 2 presents a quantitative comparison between the method of this invention and the state-of-the-art methods on the Stylized-OpenImages dataset. Tables 1 and 2 show the quantitative results on the Stylized-MSCOCO and Stylized-OpenImages datasets, verifying the effectiveness of the method of this invention in stylized image completion. On the Stylized-MSCOCO dataset, this invention first evaluates the completion effect of segmentation masks. Compared with PowerPaint (PP), the CLIP score (CS) of this invention is improved by 0.74, reflecting the role of the self-attention mask strategy, which is mainly used to ensure structural realism and indirectly preserve the shape and layout of objects. The image reward (IR) of all methods is negative, which may be because the stylized benchmark is constructed based on style transfer technology, causing the image to deviate from the original distribution. Nevertheless, the image reward (IR) is still a valuable comparative reference. The IR of this invention is improved by 0.05 compared with stable diffusion completion, indicating that the mask adjustment key value strategy can effectively capture and convey style information and improve visual consistency. Even when using bounding box masks (which lack shape information for the completed content), this invention still demonstrates robustness. The self-attention masking strategy allows the diffusion model to interpret the mask at the feature level, preserving structural authenticity and achieving good completion results even with minimal structural guidance information. Table 2 presents a similar trend, further demonstrating the effectiveness of this method under different styles and mask types.

[0083] Table 1

[0084]

[0085] Table 2

[0086]

[0087] 2. Ablation test

[0088] 2.1 Component Analysis

[0089] To verify the role of each component in this method, quantitative ablation experiments were first conducted (see Table 3). After introducing the self-attention masking strategy, the CLIP score improved by 0.92, indicating that this strategy significantly improved the structural fidelity of the masked region. After its addition, the CLIP score further improved by 2.73, highlighting its crucial role in aligning the generated content with the text prompts. The mask adjustment key value strategy helps achieve style consistency, improving the Image Reward (IR) by 0.2 and the Aesthetic Score (AS) by 0.1. Table 3 shows the quantitative ablation study results for different model variants.

[0090] Table 3

[0091]

[0092] This invention utilizes qualitative ablation experiments, such as... Figure 4 As shown, these conclusions are further validated. Without a self-attention masking strategy, the model struggles to interpret the masked regions, leading to incomplete or spatially inconsistent completion results. After removal, without semantic guidance, attention shifts to the unmasked regions, causing the generated content to mismatch with the prompts. Not using a mask-adjusting key-value strategy results in style fragmentation, preventing the completed content from integrating with its surroundings. These results demonstrate that the core components of this technique complement each other, enabling the generation of semantically aligned and visually harmonious images.

[0093] 2.2 The effect of the self-attention masking strategy

[0094] To verify the role of the self-attention masking strategy in structural fidelity, this invention divides the self-attention map into three regions, corresponding to three interaction relationships between the object and the background. Experimental results are shown below. Figure 5 Masking background-background interactions has almost no impact on background generation because the input image and noise map already provide sufficient prior background information, but it does nothing to help with object generation. Conversely, masking object-object interactions severely degrades object quality—due to the lack of strong prior information, generation relies more on object-text and object-object relationships. Without object-object attention, the generated content may retain the approximate object shape, but structural details are lost. In contrast, the strategy proposed in this invention only masks object-background interactions, effectively suppressing background interference while ensuring object fidelity.

[0095] 2.3 Hyperparameter Analysis

[0096] To further explore the impact of style harmony, Figure 6 Showing different The completion result under the given value. When When =0 (i.e., mask adjustment key-value strategy is not enabled), the 'cake' in the mask area appears close to the real style. With... As the size increases, the style of the 'cake' gradually becomes consistent with the unmasked area. Visual harmony is achieved at a value of 1.4. Beyond this value, the self-attention layer overemphasizes style information, interfering with content generation. With... As it increases, the stylistic harmony gradually improves, but when When the score exceeds 1.4, the quality of the completed content decreases significantly.

[0097] This invention is achieved through Figure 7 The parameters were investigated The effect on structural fidelity is shown, illustrating the completion results under different values. Experiments demonstrate that when... When using a hard mask, noticeable boundary artifacts appear. Within the range of 0.1-0.6, high-quality and controllable completion results are generated, achieving the optimal balance. However, when... In some cases, completion often fails, possibly because the mask is overly softened, blurring the boundary between the masked and unmasked areas.

[0098] 2.4 General Image Completion

[0099] Although this invention primarily targets stylized image completion, it also performs excellently in general image completion tasks, such as... Figure 8 As shown. Unlike stylized images, natural images do not have a clear artistic style, but to achieve visual harmony, it is still necessary to handle the consistency of lighting and color. Therefore, this invention will... Setting the value to 0.8 allows the diffusion model to capture subtle visual attributes while reducing the interference of unmasked regions on style. The completion results match the text description, maintaining a realistic and consistent visual effect, demonstrating the versatility of the method. Notably, previous methods struggled to adapt to both stylized and natural image completion tasks simultaneously, while this invention achieves excellent results in both scenarios without additional modifications.

[0100] In a preferred embodiment, the basic completion model can be replaced with a better iterative model to further improve the overall technical effect without changing the core design logic of the present invention.

[0101] The model of this invention includes multiple hyperparameters, and different styles of input images have different requirements for hyperparameter configuration. To adapt to diverse image styles and improve the model's scene adaptability, in a preferred embodiment, a hyperparameter adaptive optimization network can be added. This network analyzes the style features of the input image in real time, such as color distribution, texture structure, and composition style, and dynamically generates accurate hyperparameter configurations for images of different styles, achieving a deep match between model performance and image style.

[0102] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training-free image harmonic completion method, characterized in that, Includes the following steps: S1: Obtain the masked input image, the corresponding completion mask, and the text prompt; S2: Input the input image, the corresponding completion mask, and the text prompt into the pre-trained stable diffusion model; S3: Based on the completion mask, the self-attention map is divided into interactions within the masked region, interactions within the unmasked region, and interactions between the masked and unmasked regions. Selective weighting adjustments are made to the interactions within the masked region and the interactions between the unmasked regions to generate a reweighted self-attention map. S4: Adjust the keys and values ​​in the self-attention mechanism to introduce style features of the unmasked region into the masked region, ensuring style consistency between the masked and unmasked regions; S5: Based on the reweighted self-attention map and style coordination optimization, complete the image completion and output a harmonized completed image.

2. The training-free image harmonic completion method as described in claim 1, characterized in that, In step S3, the completion mask is adjusted to the same dimension as the self-attention map and flattened into a one-dimensional vector; the interaction within the masked region and the interaction within the unmasked region are calculated using the one-dimensional vector, and the interaction between the two types of regions is readjusted; the mask is smoothed using a soft masking mechanism, and the smoothing intensity of the mask is controlled by a smoothing factor.

3. The training-free image harmonic completion method as described in claim 1, characterized in that, In step S4, the mean of the keys and values ​​of the unmasked regions is extracted; the original keys and values ​​of the masked regions are replaced with the mean of the keys and values; a new self-attention map and latent features are calculated based on the updated keys and values, and the intensity of style transfer is controlled using hyperparameters.

4. The training-free image harmonic completion method as described in claim 1, characterized in that, It also includes step S6: applying attention-guided loss, which extracts a cross-attention map during the denoising process and calculates the loss of alignment with the text prompt token within the masked region to enhance the consistency between the completed content and the text prompt.

5. The training-free image harmonic completion method as described in claim 4, characterized in that, The calculation of the attention guidance loss includes: extracting cross-attention maps with resolutions of 16 and 32 at each time step; minimizing the difference between the target attention and the predicted attention within the mask by using a loss function, so that the attention of the text prompt token is focused on the mask region.

6. The training-free image harmonic completion method as described in claim 1, characterized in that, The denoising process of the pre-trained stable diffusion model is divided into two stages: the first stage starts from time step 0 to... The focus is on improving the structural authenticity of the supplementary content; The second phase starts with time steps Up to T, the focus is on enhancing the stylistic harmony between masked and unmasked areas; among which These are hyperparameters used to control the phase division.

7. The training-free image harmonic completion method as described in claim 1, characterized in that, The pre-trained stable diffusion model is an image generation model that does not require additional training, and all optimization steps are applied during the inference process.

8. The training-free image harmonic completion method as described in claim 1, characterized in that, The self-attention masking strategy and style reconciliation optimization are implemented separately in the encoder and decoder of the U-Net architecture to optimize feature propagation and style transfer.

Images