Training-free image harmony completion method

Abstract

The invention provides a training-free image harmonious completion method, and relates to the technical field of computer vision, and the method comprises the steps: obtaining an input image with a mask, a corresponding completion mask, and a text prompt word; inputting the input image, the complementation mask and the text cue word into a pre-trained stable diffusion model; dividing the self-attention map into interaction in a mask area, interaction in an unmasked area and interaction between the mask and the unmasked area based on the completion mask, and performing selective weighting adjustment on the interaction between the mask and the unmasked area to generate a reweighted self-attention map; adjusting keys and values in the self-attention mechanism to introduce the style features of the unmasked area into the masked area to ensure the style consistency of the masked area and the unmasked area; and completing image completion based on the reweighted self-attention map and style coordination optimization, and outputting a harmonious completion image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a training-free image harmonization completion method. BACKGROUND

[0002] With the development of digital image processing technology, image completion, as an important branch of digital image processing technology, has been widely applied in the fields of digital art, design, personalized advertising, etc. The core task is to fill in the missing areas marked by masks in the image, and to make the completed content consistent with the original image in structure and style. The current image completion technology has experienced the development from traditional texture filling to deep learning model-based, and the technologies at each stage have shown different characteristics and limitations.

[0003] Traditional texture filling technology fills the missing areas by searching for similar texture blocks in the image. Such technology can only generate general textures, cannot generate customized content according to specific requirements such as text description, and has poor completion effect on complex structure areas, making it difficult to meet the fine-grained requirements.

[0004] The completion technology based on generative adversarial network (GAN) improves the diversity of generated content to some extent, but the training process is prone to mode collapse, resulting in single generated content, and it is difficult to balance structure integrity and style consistency during the completion process, limiting its application in high-quality completion scenarios.

[0005] In recent years, the completion technology based on diffusion model has become the mainstream technology in the field of image completion. Its core mechanism is based on the Markov chain process of gradually adding noise and gradually denoising: in the training stage, the model gradually adds Gaussian noise to the original image until a random noise image is formed, and records the noise features at each step; in the inference stage, the model learns to reverse the noise process through a noise predictor to gradually recover the clear image from the random noise. Compared with the traditional texture filling and the first two generations of completion technology based on GAN, this technology has significant advantages. On the one hand, combined with the multi-modal information such as text prompt words encoded by the CLIP model, it realizes text-guided customized completion. For example, the user specifies the object type and style features of the completion area through text instructions, and the model generates content that meets the expectations, greatly improving the flexibility and controllability of completion; on the other hand, thanks to the latent diffusion model architecture, it has a natural advantage in generating high-quality image details, further expanding the application space of high-resolution image completion.

[0006] Although the existing technology has made progress in the field of image completion, it still has the following core defects: Firstly, existing image inpainting techniques rely on a large amount of training and have poor universality. The fine-tuning method highly depends on training data of a specific style. If the application scenario is switched to an untrained style, the coordination between the inpainting area and the background style will significantly decrease. At the same time, the fine-tuning of such methods requires a large amount of computing resources, making it difficult to quickly adapt to new scenarios and limiting its universality.

[0007] Secondly, the generated image may have problems of insufficient structural authenticity and disordered spatial layout. The self-attention layer of SDI, CNI and other methods has limitations and cannot effectively distinguish between the masked and unmasked areas. The principal component features of the two areas are similar, and the background features interfere with the inpainting process, resulting in incomplete structure or spatial misplacement of the generated content, which seriously affects the structural rationality of the inpainting.

[0008] Finally, the style coordination is poor and the visual fragmentation problem is prominent. The BLD and CNI methods cannot effectively transfer the style of the unmasked area to the masked area. Even though the SDI method can generate content that conforms to the text description, the style and background are still inconsistent, and there is a significant style disconnection problem. SUMMARY

[0009] To solve the above technical problems, the present application provides a training-free image harmonization inpainting method, comprising the following steps: S1: obtaining an input image with a mask, a corresponding inpainting mask and a text prompt word; S2: inputting the input image, the inpainting mask and the text prompt word into a pre-trained stable diffusion model; S3: dividing the self-attention graph into intra-masked area interaction, intra-unmasked area interaction and inter-masked and unmasked area interaction based on the inpainting mask, and selectively weighting and adjusting the inter-masked and unmasked area interaction to generate a reweighted self-attention graph; S4: adjusting the keys and values in the self-attention mechanism to introduce the style features of the unmasked area into the masked area to ensure the style consistency of the masked area and the unmasked area; S5: completing image inpainting based on the reweighted self-attention graph and style coordination optimization, and outputting a harmonized inpainting image.

[0010] Further, in step S3, the inpainting mask is adjusted to the same dimension as the self-attention graph and flattened into a one-dimensional vector. The intra-masked area interaction and the intra-unmasked area interaction are calculated through the one-dimensional vector, and the interaction between the two types of areas is re-adjusted. The soft mask mechanism is used for smoothing the mask, and the smoothing factor τ is used to control the smoothing strength of the mask.

[0011] Further, in step S4, the mean values of the keys and values of the unmasked region are extracted; the original keys and values of the masked region are replaced with the mean values; and a new self-attention graph and latent feature are calculated based on the updated keys and values, wherein the strength of style transfer is controlled using a hyperparameter λ.

[0012] Further, step S6 is further included: applying an attention guiding loss, which extracts a cross-attention graph in the denoising process and calculates a loss of the text prompt token alignment within the masked region to enhance the consistency of the completed content with the text prompt token.

[0013] Further, the calculation of the attention guiding loss includes: extracting cross-attention graphs with resolutions of 16 and 32 at each time step; and minimizing the difference between the target attention and the predicted attention within the mask through a loss function to focus the attention of the text prompt token on the masked region.

[0014] Further, the denoising process of the pre-trained stable diffusion model is divided into two stages: the first stage from time step 0 to ηT focuses on improving the structural authenticity of the completed content; and the second stage from time step ηT to T focuses on enhancing the style coordination between the masked region and the unmasked region; wherein η is a hyperparameter for controlling the stage division.

[0015] Further, the pre-trained stable diffusion model is an image generation model without additional training, and all optimization steps are applied in the inference process.

[0016] Further, the self-attention mask strategy and style coordination optimization are implemented in the encoder and decoder of the U-Net architecture, respectively, to optimize feature propagation and style transfer.

[0017] In view of the defects of the prior art, the present application needs to solve the following core technical problems: How to achieve cross-style image completion without additional training or fine-tuning, improve the universality and adaptation efficiency of the method: existing methods usually rely on a large amount of training or fine-tuning, resulting in poor universality. The present invention realizes cross-style image completion without additional training or fine-tuning, thereby improving the adaptation efficiency and universality of the method, and quickly adapting to different styles and application scenarios. How to optimize the self-attention mechanism of the diffusion model to effectively distinguish between masked and unmasked regions, avoid background feature interference, and ensure the structural authenticity of the completion content (such as complete shape and reasonable spatial layout): the current diffusion model often interferes with the generation of masked regions when processing completion tasks. By optimizing the self-attention mechanism, the present invention can effectively distinguish between masked and unmasked regions, reduce background interference, ensure the structural authenticity of the completion content, and ensure that the generated shape is complete and the spatial layout is reasonable. How to use the inherent characteristics of the diffusion model to establish a background-to-completion region style transfer mechanism to achieve style coordination between the completion region and the background: existing methods have deficiencies in style coordination, and the present invention uses the inherent characteristics of the diffusion model to design a background-to-completion region style transfer mechanism to ensure that the generated completion region is coordinated with the background style, avoid style disconnection, and improve the overall visual consistency of the image. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0019] Figure 1 is the overall framework diagram of the training-free image and harmonization completion method provided by the present application.

[0020] Figure 2 is the qualitative comparison result schematic diagram of the present application and the prior art on the Stylized-COCO dataset.

[0021] Figure 3 is the qualitative comparison result schematic diagram of the present application and the prior art on the Stylized-OpenImages dataset.

[0022] Figure 4 is the qualitative comparison result schematic diagram of the ablation experiment of different components in the present application.

[0023] Figure 5 is the comparison experiment result schematic diagram of different self-attention masking strategies in the present application.

[0024] Figure 6 is the style transfer strength hyperparameter in the present application A visual diagram illustrating the impact on the generated results.

[0025] Figure 7 It is the smoothing factor in this invention. A visual diagram illustrating the impact on the generated results.

[0026] Figure 8 This is a schematic diagram of the image completion result of the method of the present invention in a general natural scene. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0028] The overall framework of this invention is based on the U-Net architecture of a stable diffusion model, aiming to solve image inpainting tasks. The model input consists of a masked original image, a completion mask, and text prompts. The mask marks the areas to be inpainted, while the text prompts provide content guidance for the inpainting process. Next, the image undergoes preliminary processing using the basic diffusion model to generate a partially inpainted image. To enhance the structural fidelity of the inpainted results, the model introduces a self-attention masking strategy and attention-guided loss to further improve inpainting accuracy. Subsequently, to ensure stylistic consistency in the inpainted results, the model adjusts... , , A masking strategy for key values ​​is employed to optimize the harmony of styles both inside and outside the mask. Ultimately, after these processing steps, the model outputs a complete restored image, achieving efficient and high-quality output in the restoration task.

[0029] Example 1 The overall framework of the present invention is as follows Figure 1 As shown. The self-attention map in the diffusion model can effectively capture the overall layout of the image. This invention further verifies the role of self-attention blocks in image completion through experiments, a point often overlooked in existing methods. Given an incomplete image, a completion mask (mask region value is 1), and text prompts, this invention uses a stable diffusion model for image completion, with results as shown. Figure 1 As shown, it can be observed that the rabbits generated by stable diffusion have incomplete bodies and do not match the geometry of their surroundings.

[0030] To analyze the root causes of the problem, this invention extracts self-attention maps from the U-Net network and performs principal component analysis. Since skip connections inevitably couple the representations of the encoder and decoder, leading to difficulties in feature unwrapping, this invention focuses on analyzing the source of information propagation in the U-Net encoder. This invention finds that the stable diffusion completion approach, which classifies masked and unmasked regions as the same principal component in the U-Net encoder, is not ideal. Existing research indicates that the U-Net encoder extracts features from noisy latent input, and the decoder uses these features to predict the image's spatial layout. When the encoder treats masked and unmasked regions as similar components, it struggles to capture the spatial relationship between the completed content and the surrounding areas, resulting in an inaccurate layout.

[0031] To address this issue, this invention proposes a self-attention masking strategy. The self-attention mechanism effectively captures the relationships between image patches. Based on this, this invention divides the self-attention map into three regions according to the completion task: interactions within the masked region (object-object, denoted as "obj-obj"), interactions within the unmasked region (background-background, denoted as "bg-bg"), and interactions between the masked and unmasked regions (object-background, denoted as "obj-bg"). By selectively masking the 'obj-bg' interactions, the method of this invention prevents background information from penetrating the object region, improving the targeting and accuracy of the completion process. This strategy reduces background interference and improves the accuracy and quality of the completed content.

[0032] First, this invention will use the given completion mask. Adjusted to match the self-attention map in each self-attention block. With consistent dimensions, a one-dimensional complete mask is obtained. H and W are the height and width of the current feature map, which are flattened into a one-dimensional vector and incorporated into it, as shown in the following formula: ;

[0033] ;

[0034] ;

[0035] in, This indicates "obj-obj" interaction. This indicates a "bg-bg" interaction, corresponding to... Figure 2 The pink and blue areas in the middle.

[0036] During the encoding phase, this invention selectively handles the "obj-bg" interaction, corresponding to... Figure 1 The gray areas in the image are masked and then processed using the improved method. Replace the original self-attention map This adjustment makes the potential features of the masked region more focused on the completed region, reducing reliance on the unmasked region. For example... Figure 2 As shown, the masked and unmasked regions have different principal components. Unlike previous methods that only distinguish between self-attention components inside and outside the mask, this invention performs a more refined division of the self-attention map to meet the needs of the completion task. This optimization enables stable diffusion completion to better distinguish between masked and unmasked regions, improving structural realism.

[0037] However, directly introducing a one-dimensional completion mask into a self-attention block may excessively hinder information exchange between masked and unmasked regions. Therefore, in practical applications, this invention employs a soft masking strategy to obtain a soft completion mask. .

[0038] ; in, This is a smoothing factor.

[0039] The structural realism of image completion depends not only on the relationship between the object and the background, but also on the alignment between the completed object and the text cue. To better apply this concept to completion tasks, this invention proposes an attention-guided loss method—this method focuses on tokens associated with the cue within the masked region. The specific steps are as follows: At each time step, the invention performs a denoising process and extracts cross-attention maps at resolutions of 16 and 32 (these two resolutions capture the richest semantic information). Subsequently, the average cross-attention map is calculated at each resolution. , where L represents the number of tokens in the text prompt. For the i-th token, Incorporating its attention map, the formula is as follows: ; in, This represents the attention weight of the i-th token in the average cross-attention map to the j-th image patch containing the noisy latent variable. This represents the attention weight of the i-th token in the cross-attention graph corresponding to the soft-painted mask region on the j-th image patch of the noisy latent variable.

[0040] Next, enumerate the attention graphs. And define the following loss target : ; By setting the attention to the unmasked region to zero, and then minimizing the difference loss between the target attention and the predicted attention within the mask, the token attention can be concentrated as much as possible in the masked region.

[0041] This invention treats style harmony enhancement as a local style transfer task. Traditional style transfer methods typically extract attributes such as color and texture from a reference image and apply them to the target content image. In the completion task presented in this paper, this mechanism is modified: unmasked regions serve as style references, and masked regions serve as content references. Unlike traditional style transfer across images, this invention achieves local style transfer within a single image, requiring the extraction of style information from specific regions while maintaining consistency in style and content across the entire image.

[0042] The key to the self-attention module of the U-Net decoder Sum This can be used to extract style information. Based on this, the present invention proposes a mask adjustment key-value strategy, which introduces the extracted style features into the mask area to achieve style consistency. For example, this invention utilizes soft-filling masks. right Perform an update to obtain the updated K value. : ; in, This represents the mask value corresponding to the i-th block in the soft completion mask (1 indicates that it is within the completion area, and 0 indicates that it is outside the completion area). Indicates the image number 1 The characteristics of each block represent This represents the mean value of the features of the unmasked region. The present invention believes that... The overall style features of the incomplete image are represented, and therefore, these features are used to replace the original style features of the masked region to ensure that the style of the masked region remains consistent with that of the unmasked region. This invention updates V in the same way to obtain the updated V value. : ;

[0043] And based on this, a new self-attention map is calculated. and potential characteristics : ; ; in, is a hyperparameter used to control the intensity of style transfer, and d is a scaling factor with a size of K dimensions.

[0044] The diffusion model first generates a rough outline and shape of the object, and then gradually optimizes the details during the denoising process. Utilizing this characteristic, this invention divides the denoising process into two stages and introduces hyperparameters. Total time steps Divided into two stages: and .exist In this phase, the focus is on improving the structural authenticity of the supplementary content; In this phase, the focus is on enhancing the stylistic consistency between masked and unmasked regions. This design significantly reduces the computational cost during the inference process.

[0045] Example 2 To further verify the technical effectiveness of this invention, a series of experiments were designed for analysis. In the experiments, this invention was compared with several advanced text-guided image completion methods.

[0046] I. Construction of Experimental Dataset Existing commonly used image completion datasets are mainly geared towards natural image scenes and cannot meet the needs of this invention to test the coordinated completion effect in multi-style scenes. Therefore, this invention constructs two types of stylized datasets: The Stylized-COCO dataset utilizes style transfer techniques to transform images from the MSCOCO dataset into different artistic styles. This dataset includes two types of annotations: segmentation masks and bounding box masks. Text conditions use the original image description information from the MSCOCO dataset.

[0047] Stylized-OpenImages dataset: Following the same construction process as Stylized-COCO, this dataset is formed by using the OpenImages dataset as the base image source, completing stylization transformation and mask annotation.

[0048] II. Selection of Experimental Evaluation Indicators Traditional evaluation metrics used in image completion are insufficient to adequately measure the diversity and style consistency of completion results in the scenario described in this invention. Therefore, this experiment employs the following four metrics to comprehensively evaluate the completion performance of each model. The specific evaluation metrics and their core functions are as follows: CLIP Score (CS): Used to quantify the semantic alignment between the completed content and the given text prompt. A higher score indicates a stronger match between the completed result and the text description.

[0049] Image reward (IR): Based on a model trained on human preferences, it was initially applied to text-to-image generation tasks. This invention uses it to evaluate the structural realism of completed images, i.e., the consistency between the completed region and the unmasked region in terms of spatial structure and object shape.

[0050] Aesthetic Score (AS): Calculated by a linear model trained on a real image quality scoring dataset, it is used to quantify the compatibility between the completed image and the original image in dimensions such as color tone, texture style, and artistic quality.

[0051] CLIP Maximum Mean Difference (CMMD): This evaluates the difference in feature distribution between the reference image set (original image) and the generated image set (completed image) in the CLIP embedding space by calculating the MMD (maximum mean difference) distance between them. The smaller the distance, the closer the overall distribution of the completed image is to the original image, and the more stable the completion effect.

[0052] III. Comparison with state-of-the-art methods 1. Qualitative assessment The test results for the completion task are shown below. Figure 2 and Figure 3 On the Stylized-COCO dataset, this invention observes that BLD and CNI typically generate spatially inconsistent results. For example, Figure 2 In the first line, although the completed content matches the text description, the cat's shape and outline are inconsistent with the surrounding environment. This inconsistency stems from the fact that these methods fail to fully utilize the self-attention mechanism during denoising—in a self-attention map, masked and unmasked regions are treated as the same principal component, thus limiting the diffusion model's ability to model spatial relationships. In contrast, this invention introduces a self-attention masking strategy into the U-Net encoder, effectively overcoming this limitation and significantly improving structural realism.

[0053] The comparative method has made significant progress in terms of style harmony. For example, Figure 2 In the second row, BLD and CNI generate realistic horses in the masked region, but the style of the completed region does not match the unmasked region, resulting in visual inconsistency. In contrast, this invention maintains stylistic consistency between the masked and unmasked regions while aligning the cue words. This achievement is attributed to the mask adjustment key-value strategy proposed in this invention, which extracts style information from the unmasked region and integrates it into the masked region. On the Stylized-OpenImages dataset, this invention also achieves consistent completion results, validating its effectiveness across different styles and scenes.

[0054] Table 1 presents a quantitative comparison between the method of this invention and the state-of-the-art methods on the Stylized-COCO dataset; Table 2 presents a quantitative comparison between the method of this invention and the state-of-the-art methods on the Stylized-OpenImages dataset. Tables 1 and 2 show the quantitative results on the Stylized-MSCOCO and Stylized-OpenImages datasets, verifying the effectiveness of the method of this invention in stylized image completion. On the Stylized-MSCOCO dataset, this invention first evaluates the completion effect of segmentation masks. Compared with PowerPaint (PP), the CLIP score (CS) of this invention is improved by 0.74, reflecting the role of the self-attention mask strategy, which is mainly used to ensure structural realism and indirectly preserve the shape and layout of objects. The image reward (IR) of all methods is negative, which may be because the stylized benchmark is constructed based on style transfer technology, causing the image to deviate from the original distribution. Nevertheless, the image reward (IR) is still a valuable comparative reference. The IR of this invention is improved by 0.05 compared with stable diffusion completion, indicating that the mask adjustment key value strategy can effectively capture and convey style information and improve visual consistency. Even when using bounding box masks (which lack shape information for the completed content), this invention still demonstrates robustness. The self-attention masking strategy allows the diffusion model to interpret the mask at the feature level, preserving structural authenticity and achieving good completion results even with minimal structural guidance information. Table 2 presents a similar trend, further demonstrating the effectiveness of this method under different styles and mask types.

[0055] Table 1

[0056] Table 2

[0057] 2. Ablation test 2.1 Component Analysis To verify the role of each component in this method, quantitative ablation experiments were first conducted (see Table 3). After introducing the self-attention masking strategy, the CLIP score improved by 0.92, indicating that this strategy significantly improved the structural fidelity of the masked region. After its addition, the CLIP score further improved by 2.73, highlighting its crucial role in aligning the generated content with the text prompts. The mask adjustment key value strategy helps achieve style consistency, improving the Image Reward (IR) by 0.2 and the Aesthetic Score (AS) by 0.1. Table 3 shows the quantitative ablation study results for different model variants.

[0058] Table 3

[0059] This invention utilizes qualitative ablation experiments, such as...Figure 4 As shown, these conclusions are further validated. Without a self-attention masking strategy, the model struggles to interpret the masked regions, leading to incomplete or spatially inconsistent completion results. After removal, without semantic guidance, attention shifts to the unmasked regions, causing the generated content to mismatch with the prompts. Not using a mask-adjusting key-value strategy results in style fragmentation, preventing the completed content from integrating with its surroundings. These results demonstrate that the core components of this technique complement each other, enabling the generation of semantically aligned and visually harmonious images.

[0060] 2.2 The effect of the self-attention masking strategy To verify the role of the self-attention masking strategy in structural fidelity, this invention divides the self-attention map into three regions, corresponding to three interaction relationships between the object and the background. Experimental results are shown below. Figure 5 Masking background-background interactions has almost no impact on background generation because the input image and noise map already provide sufficient prior background information, but it does nothing to help with object generation. Conversely, masking object-object interactions severely degrades object quality—due to the lack of strong prior information, generation relies more on object-text and object-object relationships. Without object-object attention, the generated content may retain the approximate object shape, but structural details are lost. In contrast, the strategy proposed in this invention only masks object-background interactions, effectively suppressing background interference while ensuring object fidelity.

[0061] 2.3 Hyperparameter Analysis To further explore the impact of style harmony, Figure 6 Showing different The completion result under the given value. When When =0 (i.e., mask adjustment key-value strategy is not enabled), the 'cake' in the mask area appears close to the real style. With... As the size increases, the style of the 'cake' gradually becomes consistent with the unmasked area. Visual harmony is achieved at a value of 1.4. Beyond this value, the self-attention layer overemphasizes style information, interfering with content generation. With... As it increases, the stylistic harmony gradually improves, but when When the score exceeds 1.4, the quality of the completed content decreases significantly.

[0062] This invention is achieved through Figure 7 The parameters were investigated The effect on structural fidelity is shown, illustrating the completion results under different values. Experiments demonstrate that when... When using a hard mask, noticeable boundary artifacts appear. Within the range of 0.1-0.6, high-quality and controllable completion results are generated, achieving the optimal balance. However, when... In some cases, completion often fails, possibly because the mask is overly softened, blurring the boundary between the masked and unmasked areas.

[0063] 2.4 General Image Completion Although this invention primarily targets stylized image completion, it also performs excellently in general image completion tasks, such as... Figure 8 As shown. Unlike stylized images, natural images do not have a clear artistic style, but to achieve visual harmony, it is still necessary to handle the consistency of lighting and color. Therefore, this invention will... Setting the value to 0.8 allows the diffusion model to capture subtle visual attributes while reducing the interference of unmasked regions on style. The completion results match the text description, maintaining a realistic and consistent visual effect, demonstrating the versatility of the method. Notably, previous methods struggled to adapt to both stylized and natural image completion tasks simultaneously, while this invention achieves excellent results in both scenarios without additional modifications.

[0064] In a preferred embodiment, the basic completion model can be replaced with a better iterative model to further improve the overall technical effect without changing the core design logic of the present invention.

[0065] The model of this invention includes multiple hyperparameters, and different styles of input images have different requirements for hyperparameter configuration. To adapt to diverse image styles and improve the model's scene adaptability, in a preferred embodiment, a hyperparameter adaptive optimization network can be added. This network analyzes the style features of the input image in real time, such as color distribution, texture structure, and composition style, and dynamically generates accurate hyperparameter configurations for images of different styles, achieving a deep match between model performance and image style.

[0066] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training-free image harmonic completion method, characterized in that, Includes the following steps: S1: Obtain the masked input image, the corresponding completion mask, and the text prompt; S2: Input the input image, the corresponding completion mask, and the text prompt words into the pre-trained stable diffusion model; S3: Based on the completion mask, the self-attention map is divided into interactions within the masked region, interactions within the unmasked region, and interactions between the masked and unmasked regions. Selective weighting adjustments are made to the interactions within the masked region and the interactions between the unmasked regions to generate a reweighted self-attention map. S4: Adjust the keys and values ​​in the self-attention mechanism to introduce style features of the unmasked region into the masked region, ensuring style consistency between the masked and unmasked regions; S5: Based on the reweighted self-attention map and style coordination optimization, complete the image completion and output a harmonized completed image.

2. The training-free image harmonic completion method as described in claim 1, characterized in that, In step S3, the completion mask is adjusted to the same dimension as the self-attention map and flattened into a one-dimensional vector; the interaction within the masked region and the interaction within the unmasked region are calculated using the one-dimensional vector, and the interaction between the two types of regions is readjusted; the mask is smoothed using a soft masking mechanism, and the smoothing intensity of the mask is controlled by a smoothing factor.

3. The training-free image harmonic completion method as described in claim 1, characterized in that, In step S4, the mean of the keys and values ​​of the unmasked regions is extracted; the original keys and values ​​of the masked regions are replaced with the mean of the keys and values; a new self-attention map and latent features are calculated based on the updated keys and values, and the intensity of style transfer is controlled using hyperparameters.

4. The training-free image harmonic completion method as described in claim 1, characterized in that, It also includes step S6: applying attention-guided loss, which extracts a cross-attention map during the denoising process and calculates the loss of alignment with the text prompt token within the masked region to enhance the consistency between the completed content and the text prompt.

5. The training-free image harmonic completion method as described in claim 4, characterized in that, The calculation of the attention guidance loss includes: extracting cross-attention maps with resolutions of 16 and 32 at each time step; minimizing the difference between the target attention and the predicted attention within the mask by using a loss function, so that the attention of the text prompt token is focused on the mask region.

6. The training-free image harmonic completion method as described in claim 1, characterized in that, The denoising process of the pre-trained stable diffusion model is divided into two stages: the first stage starts from time step 0 to... The focus is on improving the structural authenticity of the supplementary content; The second phase starts with time steps Up to T, the focus is on enhancing the stylistic harmony between masked and unmasked areas; among which These are hyperparameters used to control the phase division.

7. The training-free image harmonic completion method as described in claim 1, characterized in that, The pre-trained stable diffusion model is an image generation model that does not require additional training, and all optimization steps are applied during the inference process.

8. The training-free image harmonic completion method as described in claim 1, characterized in that, The self-attention masking strategy and style reconciliation optimization are implemented separately in the encoder and decoder of the U-Net architecture to optimize feature propagation and style transfer.

Images