A text-guided image processing method based on a diffusion model

Abstract

The application realizes a text-guided image processing method based on a diffusion model. The cross-attention mechanism is used to align the words in the target text with the source image, and auxiliary editing-aware keywords and source-aware cues are generated to enhance the editing effect and improve the fidelity, respectively. In the denoising process, the attention mapping between the editing-aware keywords and the source image is injected into the denoising process to improve the alignment degree of the generated image and the target text, and a soft mask is automatically generated to balance the semantics and fidelity by comparing the diffusion model prediction between the auxiliary source-aware cues and the target text. The method is the first soft-mask-based, optimization-free, text-driven image processing method, which does not require additional annotations. Extensive experiments on two datasets show the effectiveness of the method, which quickly generates high-quality, diverse image operations while achieving a better balance between semantics and fidelity.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a text-guided image processing method based on a diffusion model. Background Technology

[0002] Image processing aims to edit the attributes and semantics of a given source image, a long-standing and intriguing problem in computer vision and image processing. Traditional image processing methods require heuristics or additional human annotations to clarify what parts should be edited, and they are often limited to fixed editing types and specific image domains. Using natural language text as input to image processing methods to describe the editing target is particularly appealing because it eliminates the need for manual intervention and allows for more general editing. With the development of generative models (such as GANs, VAEs, and diffusion models) and large-scale language and multimodal foundational models (such as CLIP and BERT), text-to-image generation has made considerable progress. Recently, diffusion-based methods such as DALL·E2, Imagen, and Stable Diffusion can synthesize new, high-quality images conditioned on the target text, guided by a classifier, semantically guided, or classifier-free diffusion. However, it is important to extend the progress of text-to-image generation to text-conditioned image processing because the guided diffusion process is largely uncontrollable. Unlike the non-strict correspondence between images and text, where many images may correspond to the target text, it is much more difficult to fully express the semantics of the target text while preserving most of the unedited areas of the source image and making limited changes.

[0003] Image editing is a classic and fascinating problem in computer vision. Text-guided image editing is the simplest and most ideal way to achieve editing intent. The biggest challenge in this task is how to achieve text-guided intent while maintaining fidelity to the original image, thus achieving an ideal trade-off. Text-conditional image processing based on diffusion models presents the challenge of balancing textual semantic representation with source image fidelity. Existing methods typically rely on text descriptions corresponding to the source image or manually created masks to maintain fidelity. However, such additional annotations are not always available, and manually labeling the corresponding source text or mask for each input image incurs high manual costs. Some methods automatically generate hard masks. However, the modification area is limited, which may lead to unrealistic or semantically inconsistent generated images. Optimization-based methods usually require fine-tuning the diffusion model or optimizing the embedding, making the operation costly and time-consuming.

[0004] Recently, diffusion-based models have begun to outperform generative adversarial networks (GANs) when performing image synthesis using simpler training settings. Early diffusion-based methods still utilize CLIP as a sampling guide for cross-modal semantic alignment. To maintain fidelity, two main approaches have been explored in diffusion models: sampling-based and optimization-based methods. Sampling-based methods focus on noise correction. For example, DiffusionCLIP and DiffEdit find useful noise through a noise deterministic DDIM process using noisy versions of the source image, such as SDEdit and LDEdit samples. CycleDiffusion uses a reconstructible encoder with a stochastic diffusion probability model for image-to-image transformation. Optimization-based methods maintain fidelity by fine-tuning the diffusion model or optimizing a design-specific objective function. UniTune fine-tunes the diffusion model to preserve details of the source image. Imagic optimizes the target text embedding and then uses the target text embedding to fine-tune the diffusion model sequentially. Pix2pix-zero optimizes an objective function for cross-attention map reconstruction to guarantee fidelity of the edited image. While these methods achieve an impressive performance balance between source image fidelity and the editing expectations of the target text, they are hampered by time-consuming fine-tuning or additional annotations (manual hints or masking). The present invention utilizes an attention attribution map of a diffusion model to find auxiliary source-aware cues and auxiliary edit-aware keywords, enhances the editing targets in the text with attention injection, and achieves fast and efficient image editing through comprehensive cue guidance.

[0005] The technical problem addressed by this invention is that extending the progress of text-to-image generation to text-conditional image processing is a challenging task. This is because guiding the image processing process under text conditions is largely uncontrollable. Unlike the non-strict correspondence between images and text, even if many images may correspond to the target text, it remains very difficult to fully express the semantics of the target text while preserving most of the unedited areas in the source image and making limited changes. Text-conditional image processing based on diffusion models faces the challenge of balancing text semantic expression and source image fidelity. Current methods typically rely on text descriptions corresponding to the source images or manually created masks to maintain image fidelity. However, such additional annotations are not always available, and manually labeling the corresponding source text or mask for each input image results in extremely high manual costs. Some methods attempt to automatically generate hard masks, but such automatically generated masks may limit the model's editing capabilities, leading to unrealistic or semantically inconsistent generated images. Optimization-based methods typically require fine-tuning the entire diffusion model or optimizing the embedding, making the operation costly and time-consuming. Therefore, these methods are not an efficient solution. Summary of the Invention

[0006] To this end, this invention first proposes a text-guided image processing method based on a diffusion model. It employs pre-trained Stable Diffusion to edit a given source image X according to the expected edit described in the target text P, resulting in an output edited image X. e Not only does it conform to the text P, but it also preserves the details of the source image X to the greatest extent possible;

[0007] The method consists of three modules: an auxiliary prompt generation module, a comprehensive prompt guidance module, and an editing-aware attention injection module;

[0008] Specifically: the auxiliary prompt generation module performs fine-grained cross-attention between the input target text and the source image, outputting auxiliary editing-aware prompts by selecting words with low attention scores to enhance editing-related information, and outputting auxiliary source-aware prompts by selecting words with high attention scores; the comprehensive prompt guidance module inputs auxiliary source-aware prompts and target text to balance the fidelity of the source image and the alignment of the target text in the output image; the editing-aware attention injection module enhances the influence of editing-aware keywords, thereby better outputting images that meet user needs.

[0009] The auxiliary prompt generation module uses an attention mechanism for text-image attribution analysis, employing a pre-trained text encoder τ. θ Set the text prompt P = {w1,…,w K} Convert to text embedding c = τ θ (P)=[c sot ,c1,…,c K ,c eot ,…],in <sot>and <eot>Insert w at the beginning and end of P respectively. k Let c be the k-th word, where K is the number of words in the text. k For w k The word embeddings are used as a condition for the diffusion model by using the following cross-attention layer:

[0010]

[0011]

[0012] in, To implement ∈ for U-Net at time step t θ The intermediate representation of the i-th layer, These are linear layer projections, Defined as a cross-attention map, through upsampling Then, by summing the results, we can obtain the diffusion attention attribution map between the k-th token and the source image X.

[0013] Stable Diffusion is used to perform forward and denoising diffusion on the source image X. θ (z t (t) is a diffusion model with parameter θ, implemented using different U-Net layers at different time steps. θ To obtain intermediate representation In extracting the embedding c = τ θ (P) After that, cross-attention as defined above is performed. Embedding words in c and fine-grained intermediate representation Alignment, through upsampling Summation yields diffuse attention attribution maps

[0014] The correlation between the k-th token and the source image is determined as follows:

[0015]

[0016] Define w as the set of indices corresponding to all nouns in the target text, and then calculate index k. max Its corresponding γ k In all It is the largest among nouns. Define a set of terms that are most relevant to the source image. As an index set to assist in editing perceptual words, the generic cue template "A photo of A [keyword]" is used to generate auxiliary source perceptual cues. Get source enhanced text embedding as

[0017] The integrated prompting guidance module, during the diffusion sampling process, uses the auxiliary source-perceived text as a condition and guides the model by modifying some of the predicted noise. Specifically, it calculates a soft mask M, which guides the model to pay attention to the source-perceived prompt information. First, it calculates the difference between the estimated noise between the target text and the auxiliary source-perceived prompt as a condition (dif). t =|∈ θ (z t ,c tgt )-∈ θ (z t ,c src )|), to obtain the mask M, then fix the difference between 0 and m×mean(dif t The values ​​are normalized to their maximum and minimum values, and finally binarized using a default threshold of 0.5. Here, m is a hyperparameter; when M = 1... h×w When, it indicates all parts that need to be modified;

[0018] The comprehensive prompts and guidance are defined as follows:

[0019]

[0020]

[0021]

[0022] The above formula will embed the source-enhanced c src and target text embedding c tgt Combined, the function IG(z) t ,c tgt ,c src ) is an integrated prompt and guidance, with the parameter λ controlling the degree of modification.

[0023] The edit-aware attention injection module records the attention attribution maps of the previous T' time steps in the denoising process, using the target text as a condition. It then injects these maps into the current denoising step, weighting them according to a certain factor to assist in the attention attribution map of the edit-aware words. This provides clues about possible edit regions in the source image, injects the attention attribution map into the attention map, and uses their correlation coefficient γ k The weighted attention injection process can be represented as

[0024]

[0025] Where e is a hyperparameter, [:,:,k] represents taking a slice from the given height and width dimensions of k, and finally performing comprehensive cue guidance and edit-aware attention injection.

[0026] The technical effects to be achieved by this invention are as follows:

[0027] (1) The text-guided image processing method based on the diffusion model proposed in this invention is the first text-conditional image processing method based on the diffusion model and soft masking without fine-tuning. It can automatically perform various complex semantic edits on any given source image without requiring any additional annotations.

[0028] (2) This invention proposes a simple and effective method for generating auxiliary prompts, which achieves fine-grained alignment of words in text and images through cross-attention. The auxiliary edit-aware keywords originate from words unrelated to the source image, thus indicating the required edits with high probability. The auxiliary source-aware prompts can represent information corresponding to the source image. In this way, we decouple the input text into prompts, which are used to guide fidelity and editing respectively.

[0029] (3) By injecting attention attribution maps and integrating target cues and auxiliary source perception cues, this invention designs a conditional denoising diffusion process to achieve guidance.

[0030] (4) Extensive experiments on two benchmark datasets demonstrate the effectiveness and efficiency of the present invention across different types of operations, including style transfer, attribute editing, and object modification. This demonstrates the effectiveness and potential of the method in text-guided image processing tasks.

[0031] (5) This invention considers both the alignment of the target text and the fidelity of the source image. Furthermore, unlike most existing text-based image editing methods, this invention does not require training or fine-tuning and does not have a separate mask generation process, making it faster. Attached Figure Description

[0032] Figure 1 This is the overall framework of the text-guided image processing method based on the diffusion model proposed in this invention.

[0033] Figure 2 This is a flowchart illustrating the comprehensive prompting and guidance process proposed in this invention.

[0034] Figure 3 These are the hyperparameter experimental results of the method proposed in this invention. Detailed Implementation

[0035] The following are preferred embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0036] This invention proposes a text-guided image processing method based on a diffusion model.

[0037] The goal of text-conditional image processing is to edit a given source image X according to the expected edit described in the target text P, so that the output edited image X is... e Not only does it match the text P, but it also preserves the details of the source image X to the maximum extent. The method of this invention achieves this goal using pre-trained Stable Diffusion without additional training. The method consists of three modules: an auxiliary cue generation module, a comprehensive cue guidance module, and an edit-aware attention injection module. The auxiliary cue generation module performs fine-grained cross-attention between the target text and the source image. The edit-aware cue enhancement strengthens editing-related information by selecting words with low attention scores. The source-aware cue enhancement improves the fidelity of the source image and enhances information relevant to the source image. The edit-aware attention injection module enhances the influence of edit-aware keywords, thereby better guiding the user to make the desired edits.

[0038] Auxiliary prompt generation:

[0039] To provide some necessary insights into image generation diffusion models conditioned on target text, text-image attribution analysis was performed using an attention mechanism. The text cue P = {w1,…,w} K Using a pre-trained text encoder τ θ Convert to text embedding c = τ θ (P)=[c sot ,c1,…,c K ,c eot ,…],in <sot>and <eot>Insert it at the beginning and end of P respectively. k Let c be the k-th word, where K is the number of words in the text. k For w k The word embeddings are used as a condition for the diffusion model by employing the following cross-attention layer:

[0040]

[0041]

[0042] in, To implement ∈ for U-Net at time step t θ The intermediate representation of the i-th layer, These are linear layer projections, Defined as a cross-attention map. This is achieved through upsampling. Then, by summing the results, we can obtain the diffusion attention attribution map between the k-th token and the source image X.

[0043] Based on experience, nouns in the target text usually refer to the main objects or scenes in the original image, such as... Figure 1 The text in the image refers to a cat. The rest of the text typically describes the desired modifications to the source image, such as "wearing sunglasses." When text is used as a control in image generation, the fidelity of the generated image to the original image is positively correlated with the clarity of the text description of the source image.

[0044] To generate auxiliary source-aware cues and auxiliary edit-aware words from the target text P, we use Stable Diffusion to perform forward and denoising diffusion on the source image X. θ (z t (t) is a diffusion model with parameter θ, implemented using different U-Net layers at different time steps. θ To obtain intermediate representation In extracting the embedding c = τ θ After (P), we perform the cross-attention defined above. Embedding words in c and fine-grained intermediate representation Alignment. We achieve this through upsampling. Summation yields diffuse attention attribution maps

[0045] The correlation between the k-th token and the source image is determined as follows:

[0046]

[0047] We assume that the correlation between the k-th label and the source image can be derived from its corresponding γ. k The value is deduced. γ k A smaller γ value indicates a lower relevance between the token and the source image, requiring more attention to execute the editing objective. Conversely, a larger γ value indicates a lower relevance between the token and the source image, requiring more attention to achieve the editing objective. k The value indicates that the token contains more information about the source image and can be used to improve the fidelity of the generated image. We define w noun as the set of indices corresponding to all nouns in the target text, and then calculate index k. max Its corresponding γ k In all It is the largest among nouns. This represents the noun most relevant to the source image. We define a set. As an index set of edit-aware words, these edit-aware words have low relevance to the source image but are more important for editing. We observed individual words... The generated image has very low fidelity to the source image. To address this issue, we use the generic cue template "A photo of A [keyword]" to generate an auxiliary source-aware cue. We obtain the source enhanced text embedding as

[0048] Comprehensive prompts and guidance module:

[0049] Cue-guided instruction is a classifier-free guidance method that can adjust the trade-off between image fidelity and text expressiveness during image generation. The general form of the classifier-free guidance formula is as follows:

[0050]

[0051] Where c tgt Indicates the embedding of the target text. This represents the embedding of an empty string. In this module, we propose a novel method that utilizes auxiliary source-aware cues and target text as guidance. For example... Figure 2 As shown, to improve the fidelity of the generated image to the source image, we condition the model with auxiliary source-aware text during the diffusion sampling process, guiding the model by modifying some of the predicted noise. To achieve this, we compute a soft mask M that guides the model to pay attention to the source-aware cue information. First, we calculate the difference between the estimated noise between the target text and the auxiliary source-aware cue as the condition (dif). t =|∈ θ (z t ,c tgt )-∈ θ (z t ,c src )|), to obtain the mask M. Then fix the difference between 0 and m×mean(dif) t Between 0 and 1, the values ​​are normalized to their maximum and minimum values, and finally binarized using a default threshold of 0.5, where m is a hyperparameter. This mask M enables the model to distinguish between the parts of the image that should be preserved and those that should be modified during editing. When M=1 h×w When, it indicates all parts that need to be modified.

[0052] Our comprehensive tips and guidelines are defined as follows:

[0053]

[0054]

[0055]

[0056] During the generation process, we embed the source enhancement c using the above formula. src and target text embedding c tgt Combined. Function IG(z) t ,c tgt ,c src ) is an integrated prompting guide, it comes from ∈ θ (z t ,c src Select a portion of the predicted noise to improve the fidelity of the source image. Another portion of the predicted noise is selected to modify the generated image, where the parameter λ controls the degree of modification. Comprehensive cue guidance balances the fidelity of the source image and the semantic expression of the target text by enhancing source-related information while preserving the semantic meaning of the target text.

[0057] Edit the perceptual attention injection module:

[0058] Some studies have demonstrated that increasing the attention weight of specific words can guide the diffusion sampling process towards those words. To improve the semantic representation effect of word-aware editing assistance, we introduce attention attribution maps into the denoising process. Using the target text as a condition, we record the attention attribution maps for the previous T' time steps of the denoising process, then inject them into the current denoising step, weighting them according to a certain factor. [Attention attribution map of word-aware editing assistance] This provides clues about potential edit regions in the source image. To better utilize these clues to guide the editing process, we inject attention attribution maps into the attention maps and use their correlation coefficients γ. k Weighted. The attention injection process is as follows: Figure 1 As shown in (c), it can be represented as

[0059]

[0060] Here, e is a hyperparameter, and [:,:,k] represents taking a slice from the given height and width dimensions k. By injecting an attention map, potential editable regions can be focused on, and the required editing information can be better communicated to the denoising model.

[0061] Finally, this invention performs integrated prompting and attention injection based on edit awareness. In this way, it considers both the alignment of the target text and the fidelity of the source image. Furthermore, unlike most existing text-based image editing methods, this invention requires no training or fine-tuning and has no separate mask generation process, making it faster.

[0062] We evaluate the performance of our method on two datasets: the COCO dataset and the EditBench dataset. Following the experimental setup established in previous work, we process images and queries from the COCO dataset to create a variety of complex image editing tasks, including object addition or deletion, color changes, etc. For evaluation purposes, we randomly select 500 image-text pairs from the processed COCO dataset. For the EditBench dataset, we select images generated by Imagen in pairs that share one of the same attributes, and then create 389 pairs consisting of a source image, target text, and target image. To ensure a fair comparison, we only collect the source image, target text, and target image; if the model requires source text, we provide an empty string as input.

[0063] We employ a 20-step DPM solver sampling method with an order of 2 and a random seed of 42. The encoding ratio parameter r is set to 0.6, T′ is set to 1, and other parameters are the default values ​​for Stable Diffusion v1.4. We inject edit-aware attention into the first 50% of the time steps of the denoising process.

[0064] In semantic image editing, it is crucial to identify the objects or scenes in the target text that require modification and to ensure that the corresponding parts in the generated image accurately reflect the parts to be modified, while preserving as many unmodified parts as possible. Using metrics established in previous work, we report the Peak Signal-to-Noise Ratio (PSNR) between the generated and source images to evaluate the fidelity of the generated image to the source image. To assess the alignment between the generated image and the target text, we report the CLIP score and CLIP similarity (CLIP sim) between the generated and target images. To assess realism, we report the Frechet Inception Distance (FID) between the generated and target images. A lower FID value indicates a more realistic image; a higher PSNR value indicates a more similar generated image to the source image. A higher CLIP score / CLIP sim indicates better alignment between the generated image and the target. To address the trade-off between CLIP score / CLIP sim and PSNR, we use the Directed CLIP Score (D-CLIP score), which provides a better balance between CLIP score / CLIP sim and PSNR. A higher D-CLIP score indicates better balance.

[0065] Table 1 presents a quantitative comparison of the COCO dataset. The results show that λ can control the degree of image editing. As λ increases, the CLIP score increases while the PSNR decreases, indicating that the generated image is more consistent with the target text but less similar to the source image. Except for PSNR, the method of this invention outperforms DiffEdit on all metrics because DiffEdit relies on hard masking, which limits the model's editing capabilities and hinders the generation of more realistic images. Compared to CycleDiffusion, the method of this invention achieves a similar D-CLIP score (i.e., a similar trade-off between CLIP score / CLIP sim and PSNR) while having a better FID value, indicating that it can generate more realistic images. Similarly, when λ = 0.8, the method of this invention outperforms StructureDiffusion on all metrics, indicating that it achieves a better semantic fidelity balance at an appropriate λ and improves the realism of the generated images. As shown in Table 2, EDIT achieves competitive results on most evaluation metrics, exhibiting a better trade-off between CLIP score / CLIP sim and PSNR. Overall, the method of this invention demonstrates effectiveness across all evaluation metrics.

[0066] Table 1 compares the results with other methods on the COCO dataset.

[0067]

[0068] Table 2 compares the results with other methods on the EditBench dataset.

[0069]

[0070] To compare the runtime of different methods, 500 images were randomly selected for editing in the non-fine-tuning method, and the average time was calculated. For the fine-tuning-based method, the number of training and inference steps per image was reported, with the number of training and inference steps set to 100 and 20, respectively. As shown in Table 3, the method of the present invention requires less runtime compared to previous methods. Specifically, the method of the present invention does not have a separate mask generation step, which makes it faster than methods used for mask generation.

[0071] Table 3 Comparison of running times for different methods

[0072] UniTune 7 minutes and 14 seconds (4 rounds) 4.96 seconds 438.96 seconds DiffEdit none 6.12 seconds 6.12 seconds Method of the present invention none 5.42 seconds 5.42 seconds

[0073] We randomly selected 500 images from the COCO dataset for parameter analysis. For ablation experiments involving λ, soft masking, or cue templates, we set the value of e to 0. For ablation experiments with e, we set λ to 0.7 and used a mask (m = 1).

[0074] In Table 4, the prompt template "A photo of A [keyword]" performed best and has been used in other experiments to enhance the auxiliary source-aware text.

[0075] Table 4 Comparison of different prompt templates

[0076]

[0077] Source enhancement conditions have a significant impact on visual quality. When λ takes values ​​of {0.3, 0.5, 0.7, 0.9, 1.0}, Figure 3 (a), (b) and (c) show that increasing λ leads to better semantic representation and realism, higher CLIP score and lower FID.

[0078] However, increasing λ can also lead to a decrease in fidelity of the source image, as evidenced by a lower PSNR. The choice of λ depends on the objective of improving fidelity or semantic representation.

[0079] The injection of perceptual attention is achieved through Eq.(9), where e=0 indicates that injection is not enabled. For example... Figure 3 As shown in (d), the FID value decreases as the absolute value of e increases, but when the absolute value of e exceeds a certain threshold (e.g., 100), the FID value increases significantly, indicating a decrease in the realism of the generated image. Furthermore, in Figure 3 In (e), the smaller the value of e, the lower the attention value of the words involved in editing, the higher the PSNR, the lower the CLIP score, and the lower the degree of editing. Therefore, we choose a conservative value of e = 50.

[0080] from Figure 3 (a) It can be seen that the larger the value of m, the smaller the proportion of the modified image area, and the less modified area in the mask m, resulting in a lower CLIP score, a higher proportion of the preserved portion, and a higher PSNR. Conversely, the smaller the value of m, the larger the proportion of the modified image area, and the higher the CLIP score. Figure 3 As can be seen from (b) and (c), the FID-CLIP score and PSNR-CLIP score achieve the best results when m=1. This result is better than the result without masking, indicating that using masking achieves a better balance between semantic representation and fidelity.

[0081] Experimental results show that the method of the present invention is a fast image processing method that can maintain the fidelity of the source image and the consistency with the target text without any training or manual operation.< / eot> < / sot> < / eot> < / sot>

Claims

1. A text-guided image processing method based on a diffusion model, characterized in that: Input source image and target text The output image contains information about the source image and conforms to the semantics of the target text. It uses pre-trained StableDiffusion to achieve the desired result based on the target text. The expected editing of the given source image described in the text This makes the output edited image Not only with text Consistent, and preserving the source image to the greatest extent possible. Details; The method consists of three modules: an auxiliary prompt generation module, a comprehensive prompt guidance module, and an editing-aware attention injection module; Specifically: the auxiliary prompt generation module performs fine-grained cross-attention between the input target text and the source image, outputs auxiliary editing perception prompts by selecting words with low attention scores to enhance editing-related information, and outputs auxiliary source perception prompts by selecting words with high attention scores; The integrated prompting and guidance module inputs auxiliary source perception prompts and target text to balance the fidelity of the source image and the alignment of the target text in the output image. The editing perception attention injection module enhances the influence of editing perception keywords, thereby better outputting images that meet user needs. The auxiliary prompt generation module uses an attention mechanism for text-image attribution analysis and employs a pre-trained text encoder. Text prompt Convert to text embedding ,in and Insert into respectively The beginning and the end, For the first One word, The number of words in the text. for The word embeddings are used as a condition for the diffusion model by using the following cross-attention layer: in, For U-Net in the 1st Each time step is implemented using U-Net. The The middle representation of the layer, , These are linear layer projections, Defined as a cross-attention map, through upsampling Then sum them up to get the first... Token and source image Diffusion Attention Attribution Map ; Use Stable Diffusion on the source image Perform forward and denoising diffusion. The parameter is The diffusion model, using different time steps Different layers to obtain intermediate representation In extracting the embedding Then, by performing the cross-attention defined above. Embedding words Intermediate representation with fine granularity Alignment, through upsampling Summation yields diffuse attention attribution maps , Determine the first The correlation between each token and the source image is as follows: Will A noun is defined as the set of indices corresponding to all nouns in the target text, and then the indices are calculated. Its corresponding In all It is the largest among nouns. Define a set of terms that are most relevant to the source image. As an index set to assist in editing perceptual words, the generic cue template "A photo of A [keyword]" is used to generate auxiliary source perceptual cues. Get source enhanced text embedding as .

2. The text-guided image processing method based on a diffusion model as described in claim 1, characterized in that: The integrated prompting and guidance module guides the model during the diffusion sampling process by modifying some prediction noise, using the auxiliary source perceived text and the target text as conditions. Specifically, it calculates a soft mask. It guides the model to pay attention to the source-aware cue information, first calculating the difference between the estimated noise and the target text and the auxiliary source-aware cue as a condition ( ), get mask Then limit the difference to and Then perform max-min normalization between the two values, and finally use the default threshold. Binarization is performed, where It is a hyperparameter, when When, it indicates all parts that need to be modified; The comprehensive prompts and guidance are defined as follows: The above formula will embed the source enhancement. and target text embedding Combination, function It is an integrated prompt and guide, parameters Control the extent of modifications.

3. The text-guided image processing method based on a diffusion model as described in claim 1, characterized in that: The edit-aware attention injection module records the text before the noise reduction process, using the target text as a condition. The attention attribution map for each time step is then injected into the current denoising step and weighted by a certain factor to assist in editing the attention attribution map of the perceived words. , This provides clues about possible edit regions in the source image, injects the attention attribution map into the attention map, and uses their correlation coefficients... The weighted attention injection process can be represented as in It is a hyperparameter. Indicates from the given Take a slice from the height and width dimensions, and finally perform comprehensive prompts and attention injection for editing awareness.

Images