Image Editing from Prompt to Prompt Using Cross-Attention Control

Abstract

Some embodiments are directed to editing a source image, which is generated based on processing a source natural language (NL) prompt using a large language image (LLI) model. Those embodiments edit the source image based on user interface input indicating an edit to the source NL prompt and, optionally, independently of any user interface input specifying a mask for the source image and / or independently of any other user interface input. Some embodiments of the present disclosure additionally or alternatively are directed to applying a prompt-to-prompt editing technique to a source image generated based on and approximating a real image to edit the source image.

Description

Background Art

[0001] Large language-image (LLI) models such as GOOGLE's IMAGEN have shown amazing semantic and compositional generative power and have attracted unprecedented attention from the research community and the public. These LLI models are trained on extremely large language-image datasets and use state-of-the-art image generation models such as autoregressive and / or diffusion models. These LLI models enable the generation of images conditioned on plain text, known as text-to-image synthesis. For example, these LLI models can generate realistic images reflecting a dog riding a bicycle in response to a plain text prompt of "a photo of a dog riding a bicycle". Recently, various LLI models have emerged that show unprecedented semantic generation.

[0002] Image editing is one of the most basic tasks in computer graphics and involves the process of modifying an input image by using auxiliary inputs such as labels, scribbles, masks, or reference images.

[0003] However, many LLI models do not provide simple editing means for the generated images and generally lack control over specific semantic regions of a given image (e.g., using only text guidance). For example, even a slight change in the text prompt can lead to the generation of completely different output images when using an LLI model. For example, changing "a photo of a dog riding a bicycle" to "a photo of a white dog riding a bicycle" can result in completely different generated images, such as an image with a different pose of the dog.

[0004] To avoid this, in many proposed LLI-based editing methods, the user has to explicitly mask a part of the image to be repaired and change the edited image of only the masked area while matching the background of the original image. However, the masking procedure is cumbersome (e.g., requires a large amount of user input to define the mask) and hinders fast and intuitive text-driven editing. Furthermore, masking the image content removes important structural information and is completely ignored in the repair process. Therefore, some editing functions, such as modifying the texture of a specific object, are outside the scope of repair.

[0005] A specific and intuitive way to edit an image is through text prompt(s) provided by the user. However, in previously proposed LLI-based editing methods, there may be no ability to edit the image generated through text prompt(s) or no ability to edit the image generated only through text prompt(s). SUMMARY OF THE INVENTION

[0006] Some embodiments of the present disclosure are directed to editing a source image, which is generated based on processing a source natural language (NL) prompt using a large language image (LLI) model. Those embodiments edit the source image based on a user interface input indicating an edit to the source NL prompt and, optionally, independently of any user interface input specifying a mask for the source image and / or independently of any other user interface input. More specifically, those embodiments generate an edited image that is visually similar to the source image but includes visual modifications that correspond to the edit to the source NL prompt. By doing so, various embodiments can utilize the same random seed(s) used when generating the source image, and further, can leverage the internal cross-attention maps generated when processing the source NL prompt using the LLI model to generate the source image. The cross-attention map is a high-dimensional tensor that associates pixels with tokens extracted from the prompt text. For example, various embodiments can control which pixels attend to which tokens of the edited prompt text during which diffusion steps by injecting at least some of the cross-attention maps during at least some of the iterations of the diffusion process based on the edited prompt.

[0007] Accordingly, various embodiments provide an intuitive image editing interface by editing only the text prompt used to generate the source image (also referred to herein as prompt-to-prompt editing). This enables voice-based, type-based (e.g., physical keyboard or virtual keyboard), and / or touch-based (e.g., interaction with emphasis elements, selection of alternative term(s)) input for editing the source image, eliminating the need for any specification of an image mask and / or other input(s). Such input for editing is natural, can be done with low latency, and enables various editing tasks that would be difficult otherwise. Further, the embodiments disclosed herein do not require additional and computationally expensive model training, fine-tuning, additional data, or optimization.

[0008] As a non-limiting example, assuming the source NL prompt is "a bear covered in fur looking at a bird", the source image reflects a bear covered in fur looking at a red bird and is generated using the LLI model based on processing "a bear covered in fur looking at a bird" and a random seed. Editing to the source NL prompt can include replacement of a subset of tokens of the source NL prompt with replacement token(s) (e.g., replacing "bird" with "butterfly"), addition of token(s) to the source NL prompt (e.g., adding "blue" before "bird"), and / or adjustment of emphasis of token(s) of the source NL prompt (e.g., increasing the emphasis in "fuzzy").

[0009] Embodiments can generate an edited image by processing at least a portion of the cross-attention map generated when generating the source image in at least some of the features (s) generated based on the edit to the source NL prompt, the source random seed, and the iteration of the process using the LLI model. When generating the edited image, by utilizing the cross-attention map in combination with the source random seed, an edited image that is visually similar to the source image but includes visual modifications consistent with the edit is obtained. For example, when "bird" is replaced with "butterfly", the edited image can replace the "red bird" in the source image with "butterfly", but can be very visually similar in other respects. Also, for example, when "blue" is added in front of "bird", the edited image can replace "red bird" with "blue bird", but can be very visually similar in other respects. As yet another example, when the emphasis in "covered with fur" is increased, the edited image can replace "bear" with "more fur-covered bear" (e.g., more and / or longer fur), but can be very visually similar in other respects. In particular, using the source random seed without utilizing the cross-attention map can result in an image that is not visually similar to the source image.

[0010] Some embodiments of the present disclosure are directed to applying the prompt-to-prompt editing technique disclosed herein to a source image that is generated based on a real image and edits the source image that approximates the real image. In those embodiments, the first prompt to be edited can be, for example, one specified by an input of a user interface and / or one automatically generated (e.g., using an automatic caption model). Further, in some of those embodiments, the source image is generated by generating a noise vector of the real image (e.g., using an inversion process) and processing an initial prompt to generate a source image that approximates the real image using the LLI model and the noise vector.

[0011] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are intended to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter that appear at the end of this disclosure are intended to be part of the subject matter disclosed herein.

Brief Description of the Drawings

[0012]

Fig. 1A

Fig. 1B

Fig. 2

Fig. 3

Fig. 4

Fig. 5

DETAILED DESCRIPTION OF THE INVENTION

[0013] Before referring to the figures, a non-limiting overview of various embodiments is provided.

[0014] As an example of various embodiments disclosed herein, let I be a source image generated by an LLI model (e.g., a text-guided diffusion model) using a prompt P and a random seed s. In some embodiments, attempt to edit the source image I guided only by the edited prompt P * and, as a result, the edited image I *is brought about. For example, consider a source image I generated from a prompt P "my new bicycle", and assume that the user wishes to edit the color of the bicycle, its material, or even replace it with a scooter while retaining the appearance and structure of the source image I. An intuitive interface for the user is to directly modify the text prompt P by further describing the appearance of the bicycle (e.g., adding "green" before "bicycle") or replacing it with other words (e.g., replacing "bicycle" with "scooter"). In contrast to some prior arts, various embodiments disclosed herein avoid relying on any user-defined mask (e.g., a mask defined through interaction with the source image I) to assist or represent where the editing of the source image I should be performed. For example, those various embodiments avoid relying on any user-defined mask generated based on user interaction with the source image I that defines the "bicycle" within the source image I. Further, in various embodiments disclosed herein, using the LLI model, (a) the same random seed s used when generating the source image (I) and (b) the edited text prompt P * (instead of the original text prompt) alone will result in completely different images with different structures and compositions. For example, (b) the edited text prompt P * is "my new green bicycle" (where "green" is added before "bicycle"), using (a) the same random seed s and (b) the edited text prompt P * alone may result in a generated image that includes a "green bicycle". However, compared to the source image I, such a generated image has different structures and compositions (e.g., includes different background objects (s)).

[0015] In the embodiments disclosed herein, it is recognized that the structure and appearance of the generated image depend not only on the random seed s but also on the interaction between pixels with respect to text embedding through the diffusion process. More specifically, in the embodiments disclosed herein, the image editing function from prompt to prompt is enabled by modifying the interaction between pixels and text that occurs in the cross-attention layer, and it is recognized that the structure and composition of the source image I during editing are maintained. More specifically, when generating the image I edited using the LLI model * by injecting at least some of the cross-attention maps generated when generating the source image I, it becomes possible to maintain the composition and structure of the source image I.

[0016] For additional context regarding the cross-attention map, a specific example of cross-attention in the IMAGEN LLI model (including text-conditioned diffusion models) is described in more detail. Embodiments of IMAGEN include three text-conditioned diffusion models: a 64×64 model from text to image, as well as two super-resolution models - a 64×64→256×256 model and a 256×256→1024×1024 model. These predict the noise ε θ (z t , c, t) via a U-shaped network for t in the range from T to 1, where z tis the latent vector and c is the text embedding. More specifically, the 64×64 model starts from a random noise seed and uses a U-Net. The model is conditioned on the text embedding via both cross-attention layers at the resolution of downsampling and upsampling within the U-Net and hybrid attention layers at the resolution of downsampling and upsampling within the U-Net. The 64×64→256×256 model conditions on a naively upsampled 64×64 image. An efficient version of the U-Net is used that includes a hybrid attention layer at the bottleneck (resolution of 32). The 256×256→1024×1024 model conditions on a naively upsampled 256×256 image. An efficient version of the U-Net is used that includes only a cross-attention layer at the bottleneck (resolution of 64).

[0017] In such IMAGEN LLI models, and / or other LLI models (if any), the composition and geometry are generally determined at the resolution of the output of the model from text to image / the input to the initial super-resolution model (e.g., 64×64 in the example above). Thus, in some embodiments, when generating an edited image, the super-resolution process can be used as is to perform adaptation only in the diffusion process from text to image. When generating an image using an IMAGEN LLI model, and / or other LLI models (if any), each diffusion step or iteration t involves predicting noise f from a noisy image z t and a text embedding ψ(P). At the last diffusion step, the image I = z0 generated by this process is obtained. It should be noted that the interaction between the two modalities occurs during noise prediction, where the embeddings of visual and textural features are fused using cross-attention layers that produce spatial attention maps for each text token.

[0018] More formally, the noisy image Φ(z t)'s deep - space features are projected onto the query matrix Q = l Q (Φ(z t )) and the text embedding is projected onto the key matrix K = l Q ,l K ,l V and the value matrix V = l K (ψ(P)) via the learned linear projections l V (ψ(P)). Next, the attention map is

Number

Number

[0019] Intuitively, the cross - attention output MV is the weighted average of the values V, where the weights are the attention map M that correlates with the similarity between the query matrix Q and the key matrix K. In practice, to enhance their expressiveness, multi - head attention can be used in parallel, and then the results are concatenated and passed through a learned linear layer to obtain the final output.

[0020] IMAGEN and / or other LLI model(s) condition the text prompt in the noise prediction for each diffusion step via two types of attention layers of hybrid attention that act as both self-attention and cross-attention by concatenating the text embedding sequence to the key and value pairs of i) the cross-attention layer and ii) each self-attention layer. Since various embodiments can intervene only in the cross-attention part of the hybrid attention, both of them can be referred to as cross-attention. That is, only the last channel that references text tokens is modified in the hybrid attention module.

[0021] When controlling cross-attention in the IMAGEN LLI model and / or other LLI model(s), it should be noted again that the spatial layout and geometry of the generated image depend on the cross-attention map generated when generating that image. This interaction between pixels and text can be observed from a plot of the average attention map generated when generating the image. In such a plot, it can be observed that pixels are attracted by the words that describe them. For example, in the case of a prompt containing the word "bear", it can be observed that the pixels depicting the bear are correlated with the word "bear". Such observations indicate that the structure of the image is already determined at an early step of the diffusion process.

[0022] Since the attention reflects the overall composition, the attention map M obtained from the generation of the original prompt P and the source image I using the LLI model can be injected into the second generation using the modified prompt P * when generating the edited image I * and the LLI model. This results in an edited image I *Synthesis becomes possible. Such examples are specific examples of a broader set of attention-based operations that enable different types of intuitive editing. Therefore, in the following paragraphs, a more general framework will be described, followed by details of various specific editing operations.

[0023] DM(z t , P, t, s) is defined as the calculation for a single step t of a diffusion process that outputs a noisy image z t-1 and an attention map M t (omitted if not used).

Number

Number

Number

Number

[0024] Common algorithms for controlled image generation can include running an iterative diffusion process simultaneously for both prompts, where attention-based operations are applied at each step according to the desired editing task. The internal randomness used in each of the diffusion processes, which can be reflected by a random seed(s), can be fixed / the same for each process. This is due to the nature of diffusion models where two random seeds can produce significantly different outputs even for the same prompt. Formally, our common algorithm is as follows.

[0025] More formally, the common algorithm for various embodiments can be as follows.

Number

[0026] Note that in the above algorithm, the image I generated by the prompt P and the random seed s can be defined as an additional input. Still, the algorithm will remain the same. Also note that in the above algorithm, the forward call on the 7th line can be skipped by applying the editing function inside the diffusion forward function. Additionally or alternatively, the diffusion step can be applied to both z t-1 and

Number

[0027]

Number

[0028] In word substitution, a user interface input is provided that indicates that the user has substituted one or more of the tokens of the original prompt with others. For example, if the user interface input indicates an edit to the original prompt "big red bicycle" to the edited prompt "big red vehicle", "bicycle" can be substituted with "vehicle". Such user interface input can be via touch and / or type input that deletes "bicycle" and types "vehicle", and / or via spoken user interface input (e.g., a spoken input of "replace bicycle with vehicle"). In word substitution and / or other editing operations, the challenge is to address the content of the edited prompt while maintaining the original composition. For this purpose, embodiments inject an attention map generated when generating the source image into the generation of an edited image using the edited prompt. However, the proposed injection of attention can overly constrain the geometry, especially when large-scale structure modifications such as from "bicycle" to "vehicle" are involved. Such an over-constraint of the geometry can be addressed in some embodiments of word substitution editing by softer attention constraints. For example, softer attention constraints can be represented by the following editing function: [Number]

[0029] In the aforementioned editing function, τ is a timestamp / iteration parameter, and this parameter determines up to which step the injection is applied. Note that the composition is determined at an early step of the diffusion process. Thus, by restricting the number of injection steps, the composition of the newly generated image can be guided while still allowing the degrees of freedom of the geometry necessary to conform to the new prompt. Additional or alternative conformations are to assign different numbers of injection timestamps to different tokens in the prompt. If two words are represented using different numbers of tokens, the map can be replicated / averaged as needed using an alignment function such as that described for the addition of new phrases.

[0030] For the addition of a new phrase, a user interface input indicating that the user has added new token(s) to the original prompt is provided. For example, "by children's drawings" can be added before the original prompt "a castle next to a river", in which case the user interface input indicates such a forward addition. For example, the user interface input can include typing "by children's drawings" at the beginning of the original prompt, or it can be a spoken user interface input such as "add by children's drawings, before". For the addition of a new phrase, in order to retain common details, embodiments can apply attention injection only to common token(s) from both prompts. For example, attention injection can be applied only to "a castle next to a river" in the aforementioned example. More formally, an alignment function A can be utilized, which takes a token index from the edited prompt P * and outputs the corresponding token index in P * , or outputs nothing if there is no match. In such an alignment function, an exemplary editing function can be represented as follows:

Number

[0031] It should be recalled that in the above editing function, index i corresponds to the pixel value and j corresponds to the text token. Optionally, in the above editing function, optionally and similar to the word substitution editing function, a time stamp τ can be set to control the number of diffusion steps to which the injection is applied. Such an editing function enables functions from prompt to prompt, such as styling, specifying object attributes, or global operations.

[0032] In attention reweighting, a user interface input is provided that indicates the user's desire to strengthen or weaken the degree to which the token(s) of the original prompt affect the original source image. For example, the original prompt could be "fluffy red ball", and the user may request an edited image where the ball is fluffier or less fluffy than the original image. Such a user interface input indicating an increase or decrease in fluffiness could be, for example, an interaction with a slider or up and down arrows presented along with "fluffy", bold or underlined "fluffy" and / or an utterance input (e.g., "more fluffy"). In the attention reweighting of the token(s) of the original prompt, the assigned token(s) j * corresponding to the token(s) to which the emphasis user interface input is directed are used with a scaling parameter c. For example, the scaling parameter c can be a negative parameter when the emphasis input indicates a decrease and a positive parameter when the emphasis input indicates an increase, and optionally, can have a magnitude based on the degree of increase or decrease indicated by the emphasis input. For example, the scaling parameter c can be represented as c ∈ [-2, 2]. The remaining part of the attention map can remain unchanged. Such an editing function is

Number

[0033] Next, some non-limiting examples of the practical uses of various embodiments are provided, which demonstrate that intuitive text-only editing can be enabled by controlling the spatial layout corresponding to each word in the prompt provided by the user.

[0034] One practical use is local editing of the source image without the need for any masks provided by the user, through editing the source prompt provided by the user. For example, the source image can be generated using the prompt "lemon cake" and the LLI model. The input to the user interface can replace "lemon" with "pumpkin", resulting in an edited prompt of "pumpkin cake". By utilizing the embodiments disclosed herein, an edited image can be generated that retains the spatial layout, geometry, and semantics of the source image. On the other hand, naively giving the prompt "pumpkin cake" to the synthesis model results in completely different geometry even when using the same random seed in a deterministic setting.

[0035] Other practical uses are to perform structural modifications on the source image in addition to, or instead of, modifying only the texture. For example, the source image can be generated using a prompt that includes "bicycle" (among other words, if any), and an LLI model, and the user interface input can replace "bicycle" with "car". By utilizing the embodiments disclosed herein, an edited image can be generated that changes the "bicycle" in the source image to "car" in the edited image. It has been observed that the more diffusion steps to which cross-attention injection is applied in generating the edited image, the higher the fidelity to the original image. However, the optimal result is not necessarily achieved by applying the injection throughout the entire diffusion steps. Thus, cross-attention injection can be optionally applied only to a subset of steps or iterations, such as a threshold percentage that is between 5% - 95%, 15% - 90%, or other boundaries (if any). Optionally, interactive user interface element(s) can be presented along with the edited prompt, whereby the user input can define the fidelity to the original image to be adhered to in generating the edited image. If such user interface element(s) are provided, the subset of steps or iterations to which cross-attention injection is applied can correspond to the user interface input directed to those interactive user interface element(s) (if any). For example, the interactive user interface element(s) can include a slider, and the number of iterations to which cross-attention injection is applied can be based on the position of the slider.

[0036] Another practical use is that, instead of replacing one word with another, the user may wish to add a new specification to the generated source image. For example, the generated source image can be generated based on a source prompt of "a car on the side of the street", and a user interface input can be provided to add "crushed" in front of the car, resulting in an edited prompt of "a crushed car on the side of the street". In such a case, the attention map of the source prompt can be used when generating the edited image, while it is also possible to use the newly added word ("crushed") and the corresponding attention map when generating the edited image. Thereby, while the background of the source image is still retained, an edited image including a crushed car (contrarily not included in the source image) can be obtained.

[0037] Another practical use is to perform global editing while maintaining the image composition of the source image. In such a use, the editing affects all parts of the image, but the original composition such as the position and identity of the objects should still be maintained. For example, by editing a source prompt of "a car on the side of the street" to "a car on a snow-covered street", snow can be added to the background and the car while maintaining the background and the car of the source image. As another example, by editing a source prompt of "a photo of a waterfall" to "an impressionist painting of a waterfall", the original composition of the source image can be maintained while changing it from a photo to an impressionist painting.

[0038] Various embodiments are described herein with respect to applying prompt - to - prompt editing techniques to source images that are generated by processing source prompts using an LLI model. However, embodiments of the present disclosure additionally or alternatively are directed to applying the prompt - to - prompt editing techniques disclosed herein to editing source images that are generated based on real images (e.g., real images captured by a real - world physical camera), and editing source images that approximate real images. In those embodiments, the first prompt to be edited can be, for example, one specified by user interface input and / or one automatically generated (e.g., using an automatic captioning model). Further, in some of those embodiments, the source image is generated by generating a noise vector of the real image (e.g., using an inversion process) and processing an initial prompt using the LLI model and the noise vector to generate a source image that approximates the real image.

[0039] In embodiments that apply prompt - to - prompt editing techniques to editing source images, it is recognized that editing a real image may require finding an initial noise vector that, when given to a diffusion process, results in a given input image. This process is generally known as inversion, but has not conventionally been used in LLIs such as text - guided diffusion models. A naive approach would be to add Gaussian noise to the real image and then run a predefined number of diffusion steps. However, such an approach can result in significant distortion. Accordingly, some embodiments disclosed herein employ an improved inversion approach based on the deterministic denoising diffusion implicit model (DDIM) rather than the denoising diffusion probability model (DDPM). These embodiments can run the diffusion process in the reverse direction, i.e., x T →x0 is replaced by x0→x T where x0 is set to be the real image.

[0040] Such an inversion process can yield satisfactory results. However, such an inversion is not sufficiently accurate in many other cases. This may be due in part to the distortion-editing trade-off, where reducing the guidance parameters without a classifier (i.e., reducing the influence of the prompt) improves the reconstruction but limits the ability to perform significant operations. To mitigate this limitation, in some embodiments, a mask directly extracted from the attention map is used to restore the unedited regions of the original image. Note that in those embodiments, the mask is generated without guidance from the user. Furthermore, in some of these embodiments, this approach can also function using a naive DDPM inversion scheme (adding noise and then removing the noise).

[0041] DDPM is a generative latent variable model aimed at modeling a distribution p θ (x0) that approximates the data distribution q(x0) and is easy to sample from. DDPM models the "forward process" in the space of x0 from data to noise.

[0042] This process is a Markov chain starting from x0, and noise is gradually added to the data to generate latent variables x1,…,x T ∈X. Thus, the sequence of latent variables is [Number] Following [Number], the steps in the forward process are Gaussian transitions parameterized by a schedule β0,…,β T ∈(0,1). [Number] Defined as. When T is large enough, the last noise vector x T follows an approximately isotropic Gaussian distribution.

[0043] An interesting property of the forward process is that the latent variable x can be expressed directly as the following linear combination of noise and x0 without sampling the intermediate latent vectors t : [Number] where [Number]

[0044] To sample from the distribution q(x0), a double "reverse process" p(x T |x t-1 |x t ) from data to the isotropic Gaussian noise x t-1 |x t ) is defined by sampling the posterior q(x t-1 |x t ). Since the computationally intractable reverse process q(x t-1 |x t ) depends on the unknown data distribution q(x0), it can be approximated by a parameterized Gaussian transition network [Number] . μ θ (x t , t) can be replaced by predicting the noise ε θ (x t , t) added to x0 using Equation 2

[17] .

[0045] Under this definition [Number] Bayes' theorem can be used to approximate

[0046] Given the trained ε θ (x t , t), the following sampling method can be used: x t-1 = μ θ (xt , t) + σ t z, z ~ N(0, I). The σ of each sample stage t can be controlled. In DDIM, σ t = 0 can be used to deterministically determine the sampling process. The reverse process is finally trained by solving the following optimization problem:

Equation

[0047] Here, referring to the drawings, FIG. 1A shows generating a source image 103A based on processing a natural language prompt 101A using an LLI model 150, and using the LLI model 150 to generate an edited image 107A that is visually similar to the source image 103A but includes visual corrections that match the edits to the NL prompt used to generate the source image 103A as reflected in the prompt editing input 106A. Schematic illustrations of exemplary components and interactions that may be involved are depicted.

[0048] In FIG. 1A, the client device 110 can provide an NL prompt 101A, such as an example of "a photo of an orange cat riding a bicycle". The NL prompt 101A can be generated based on user interface input provided by the user at the client device 110, such as typed input or spoken input. For example, the NL prompt 101A can be based on text from speech recognition executed based on spoken input received at the client device 110.

[0049] The source image engine 120 can process the NL prompt 101A using the LLI model 150 to generate the source image 103A. When generating the source image 103A, one or more random (true random or pseudo-random) seeds 104A can be utilized. Additionally, a cross-attention map 105A is generated when generating the source image 103A. The random seed(s) 104A and the cross-attention map 105A can be provided to the edited image engine 130.

[0050] The edited image engine 130 receives a prompt edit input 106A, which is a user interface input provided by the client device 110 and specifies one or more edits to the NL prompt 101A, such as a replacement input (e.g., replacing "bicycle" with "horse"), an addition input (e.g., adding "green" before "bicycle"), and / or an emphasis adjustment input (e.g., increasing the emphasis of "orange"). In response to receiving the prompt edit input 106A, the edited image engine 130 can interact with the LLI model 150 to generate an edited image 107A that is visually similar to the source image 103A but includes visual modifications that match the edit(s) to the NL prompt 101A reflected by the prompt edit input 106A.

[0051] When interacting with the LLI model 150 to generate the edited image 107A, the editing image engine 130 can utilize the random seed(s) 104A used when generating the source image 103A, can utilize editing features based on the edits reflected by the prompt editing input 106A (e.g., the text embedding of the modified prompt reflected by the edit), and can utilize at least some of the cross-attention maps 105A in at least some of the iterations of generating the edited image 107A. Which cross-attention maps 105A are utilized when generating the edited image 107A and / or in which iteration the cross-attention maps are utilized may depend on the type(s) of edit(s) reflected by the prompt editing input 106A (e.g., depending on whether the edit is of the replacement, addition, or emphasis adjustment type).

[0052] FIG. 1B schematically depicts exemplary components and interactions that may be involved in using the LLI model 150 to generate a source image 103B that approximates the real image 102B and using the LLI model 150 to generate an edited image that is visually similar to the source image 103B but includes visual modifications that match the edits to the NL prompt used to generate the source image 103B, as reflected by the prompt editing input 106B.

[0053] In FIG. 1B, the client device 110 can provide the real image 102B to the noise vector engine 120. The noise vector engine 120 can generate a noise vector 102B1 for the real image 102B. For example, the noise vector engine 120 can use the inversion process and the real image, such as by using the DDIM or DDPM inversion process, to generate the noise vector 102B1. The noise vector 102B1 is provided to the source image engine 120 together with the NL prompt 101B for the real image 102B. The NL prompt 101B can be provided by the client device and can be based on a user interface input (e.g., a user interface input that is a caption curated by the user for the real image 102B), and / or can be provided by a caption engine 140 that automatically generates the NL prompt by processing the real image 102B using a caption model.

[0054] The source image engine 120 can process the NL prompt 101B and the noise vector 102B1 using the LLI model 150 to generate a source image 103B that approximates the real image 102B. One or more random (true random or pseudo-random) seeds 104B can be utilized when generating the source image 103B. Further, a cross-attention map 105B is generated when generating the source image 103B. The random seed(s) 104B and the cross-attention map 105B can be provided to the edited image engine 130.

[0055] The editing image engine 130 receives a user interface input provided by the client device 110, a prompt editing input 106B that specifies one or more edits to the NL prompt 101B (optionally renderable on the client device 110 based on the output from the caption engine 140), such as replacement input, addition input, and / or emphasis adjustment input. In response to receiving the prompt editing input 106B, the editing image engine 130 can interact with the LLI model 150 when generating an edited image 107B that is visually similar to the source image 103B but includes visual modifications that match the edit(s) to the NL prompt 101B reflected by the prompt editing input 106B.

[0056] When interacting with the LLI model 150 when generating the edited image 107B, the editing image engine 130 can utilize the random seed(s) 104B used when generating the source image 103B, can utilize editing features based on the edits reflected by the prompt editing input 106B (e.g., text embeddings of the modified prompt reflected by the edit), and can utilize at least some of the cross-attention maps 105B in at least some of the iterations of generating the edited image 107B. Which cross-attention maps 105B are utilized when generating the edited image 107B and / or in which iteration the cross-attention maps are utilized may depend on the type(s) of edit(s) reflected by the prompt editing input 106B (e.g., depending on whether the edit is of the replacement, addition, or emphasis adjustment type).

[0057] Figure 2 shows an exemplary method 200 for generating a source image based on processing a natural language prompt using an LLI model, and storing the random seed(s) used in the processing and the cross-attention maps resulting from the processing. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system can include various components of various computer systems, such as one or more components of a server computing device(s). Further, the operations of method 200 are shown in a particular order, but this is not meant to be limiting. One or more operations can be reordered, omitted, or added.

[0058] In block 202, the system receives a natural language prompt. For example, the natural language prompt can be provided based on a user interface input at a client device, such as a user interface input directed to an interface or front end of the system that is accessible via the client device.

[0059] In block 204, the system generates one or more source random seeds. For example, the system can use a random process or a pseudo-random process to generate the source random seed(s).

[0060] In block 206, the system generates a source image based on processing the source random seed of block 204 and the NL prompt of block 202 using an LLI model. When generating the source image based on the processing using the LLI model, cross-attention maps are generated as described herein. The cross-attention maps can include values that associate the tokens of the NL prompt with the pixels of the generated source image.

[0061] In block 208, the system stores the random seed(s) of block 204 and the cross-attention map generated during the generation of the source image in block 206 (e.g., at least temporarily in memory).

[0062] In block 210, the system causes the rendering of the source image and the NL prompt 210. For example, the system can cause such rendering on a client device that provided the natural language prompt of block 202.

[0063] FIG. 3 shows an exemplary method 300 of generating an edited image that is visually similar to a source image but includes visual corrections that match the edits to the NL prompt used to generate the source image using an LLI model. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system can include various components of various computer systems, such as one or more components of a server computing device(s). Further, although the operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations can be reordered, omitted, or added.

[0064] In block 302, the system receives a user interface input indicating an edit to the source NL prompt used to generate the source image. The user interface input can be received on a client device in response to the rendering of the source image and optionally in response to the rendering of the NL prompt used to generate the source image. The source image can be the source image of block 206 of the iteration of method 200 of FIG. 2, and the NL prompt can be the NL prompt of block 206 of the iteration of method 200 of FIG. 2. Alternatively, the source image can be the source image (approximating a real image) of block 410 of the iteration of method 400 of FIG. 4, and the NL prompt can be the NL prompt of block 404 of the iteration of method 400 of FIG. 4.

[0065] In various embodiments, block 302 includes one or more of sub-blocks 302A, 302B, and 302C. In sub-block 302A, the user interface of the block 302 input includes replacement inputs. The replacement inputs can reflect an edit that is a replacement of a subset of tokens of the source NL prompt with one or more replacement tokens that are different from the subset of tokens of the source NL prompt. In sub-block 302B, the user interface of the block 302 input includes additional inputs. The additional inputs can reflect an edit that is an addition of one or more additional tokens to the source NL prompt. In sub-block 302A, the user interface of the block 302 input includes emphasis adjustment inputs. The emphasis adjustment inputs can reflect an edit that is an adjustment of the emphasis in one or more emphasis tokens of the source NL prompt, where the adjustment is an increase or decrease in emphasis, and optionally can reflect the magnitude of the increase or decrease.

[0066] In block 304, the system generates an edit feature based on an edit to the source NL prompt reflected by the user interface input received in block 302. For example, if the edit is a replacement, the system can generate an edit feature that includes a text embedding of a modified prompt that conforms to the source NL prompt, but replaces a subset of the tokens of the source NL prompt with the edited tokens. As another example, if the edit is an addition, the system can generate an edit feature that includes a text embedding of a modified prompt that includes the source NL prompt and the additional tokens. As yet another example, if the edit is an adjustment of the emphasis of one or more emphasis tokens, the system can generate an edit feature that includes a scaled attention map(s) for the one or more emphasis tokens.

[0067] In block 306, the system identifies the source seed(s) and cross-attention map(s) used when generating the source image of block 302. The source seed(s) and cross-attention map(s) can be those that generate the source image in an iteration of method 200 of FIG. 2, or can be those that generate the source image (approximating the real image) in an iteration of method 400 of FIG. 4.

[0068] In block 308, the system uses the LLI model to generate an edited image based on processing at least some of (A) the editing features generated based on the editing of the source NL prompt (generated in block 304), (B) the source seed(s) (identified in block 306), and (C) the cross-attention map(s) (identified in block 306).

[0069] In some embodiments, block 308 includes sub-block 308A where the system uses only a subset of the cross-attention maps and / or uses the cross-attention maps only with a subset of the iterations of processing. In some versions of those embodiments, whether a subset of the cross-attention maps is utilized or which subset of the cross-attention maps is utilized may depend on the edits to the source NL prompt reflected by the user interface input received at block 302. Further, in some of those versions, or in other versions of those embodiments, whether the cross-attention maps are applied only to a subset of the iterations and / or which subset the cross-attention maps are applied to may depend on the edits to the source NL prompt reflected by the user interface input received at block 302. For example, if the edit is a replacement, only a subset of the cross-attention maps that excludes what corresponds to the replaced token(s) can be utilized and is utilized only with a subset of the iterations. As another example, if the edit is an addition, the cross-attention maps can optionally include all of the cross-attention maps, but they can be utilized only with a subset of the iterations (e.g., not with the processing features(s) corresponding to the replacement token(s)). As yet another example, if the edit is an adjustment of emphasis in the emphasized token(s), a first subset of the cross-attention maps can be utilized for the non-emphasized token(s), and a scaled version of a second subset of the cross-attention maps can be utilized for the emphasized token(s).

[0070] In some additional or alternative embodiments, sub-block 308A can include a system that always uses the cross-attention map only for a subset of the processing iterations, such as only at the threshold percentage of the iterations. For example, the threshold can be between 5% and 95%, between 15% and 90%, between 25% and 75%, or between other boundaries (s). Optionally, in some versions of those additional or alternative embodiments, interactive user interface element(s) can be presented, whereby user input can define the fidelity to the original image to be adhered to when generating the edited image. In some of those versions, the threshold can be determined by the system based on the interaction(s) with the user interface element(s).

[0071] At block 310, the system causes the rendering of the edited image and optionally the rendering of the edited NL prompt. For example, the system can cause such rendering on the client device that provided the user interface input of block 302.

[0072] At optional block 312, the system can monitor for new user interface input that indicates further editing to the source NL prompt and that is added to the editing(s) of previous iteration(s) of block 302. If such new user interface input is detected, the system can proceed to execute other iterations of blocks 302, 304, 306, 308, and 310 based on such new user interface input.

[0073] Figure 4 shows an exemplary method 400 for generating a source image that approximates a real image by using an LLI model to process a natural language prompt for the real image and a noise vector for the real image, and storing the random seed(s) used in that processing and the cross-attention map generated in that processing. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system can include various components of various computer systems, such as one or more components of a server computing device(s). Further, although the operations of method 400 are shown in a particular order, this is not meant to be limiting. One or more operations can be reordered, omitted, or added.

[0074] In block 402, the system identifies a real image captured by a real camera, such as a real image uploaded from a client device.

[0075] In block 404, the system identifies an NL prompt for the real image. When identifying the NL prompt for the real image, the system can execute sub-block 404A or sub-block 404B.

[0076] In sub-block 404A, the NL prompt for the real image is generated based on user interface input. For example, in block 402, when the real image is received from a client device, the NL prompt can also be received and can respond to user interface input received at the client device. For example, the user interface input can be received at the client device in response to the rendering of a prompt such as "Please provide a natural language description of this image."

[0077] In sub-block 404B, the NL prompt for the real image is generated based on processing the real image using a caption model or other vision-language model.

[0078] In block 406, the system generates a noise vector for the real image. For example, the system can generate a noise vector based on applying an inversion process, such as a DDIM or DDPM inversion process, to the real image.

[0079] In block 408, the system generates one or more source random seeds. For example, the system can use a random process or a pseudo-random process to generate the source random seed(s).

[0080] In block 410, the system uses the LLI model to generate a source image that approximates the real image by processing the source random seed(s) of block 408, the noise vector of block 406, and the NL prompt of block 404. When generating the source image based on such processing using the LLI model, a cross-attention map occurs as described herein. The cross-attention map can include values that associate the tokens of the NL prompt with the pixels of the generated source image.

[0081] In block 412, the system stores (e.g., at least temporarily in memory) the random seed(s) of block 408 and the cross-attention map that occurred during the generation of the source image in block 410.

[0082] In optional block 414, the system causes the rendering of the source image and / or the NL prompt 210. For example, the system can cause such rendering on the client device that provided the real image of block 402.

[0083] FIG. 5 is a block diagram of an exemplary computing device 510 that can optionally be utilized to implement one or more aspects of the techniques described herein. For example, all or aspects of the computing device 510 can be incorporated into a server(s) or other computing device(s) that are utilized to implement the prompt-to-prompt editing techniques disclosed herein.

[0084] The computing device 510 typically includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices can include, for example, a storage subsystem 524 that includes a memory subsystem 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices enable user interaction with the computing device 510. The network interface subsystem 516 provides an interface to external networks and is coupled to corresponding interface devices within other computing devices.

[0085] Examples of the user interface input device 522 include a keyboard, a mouse, a trackball, a pointing device such as a touchpad or a graphics tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 510 or a communication network.

[0086] Examples of the user interface output device 520 include non-visual displays such as a display subsystem, a printer, a fax machine, or an audio output device. Examples of the display subsystem include a cathode ray tube (CRT), flat panel devices such as a liquid crystal display (LCD), a projection device, or other mechanisms for creating a visible image. The display subsystem can also provide a non-visual display via, for example, an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 510 to a user or another machine or computing device.

[0087] The storage subsystem 524 stores programming and data structures that provide some or all of the functionality of some of the modules described herein. For example, the storage subsystem 524 can include logic for executing selected aspects of the methods of FIGS. 2, 3, and / or 4 and logic for implementing the various components described herein.

[0088] These software modules are typically executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 can include a number of memories such as a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 can provide for the persistent storage of program files and data files and can include a hard disk drive, a floppy disk drive with an associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functions of certain embodiments can be stored in the file storage subsystem 526 within the storage subsystem 524 or in other machines accessible by the processor(s) 514.

[0089] The bus subsystem 512 provides a mechanism that enables the various components and subsystems of the computing device 510 to communicate with each other as intended. The bus subsystem 512 is shown schematically as a single bus, but multiple buses can be used in alternative embodiments of the bus subsystem.

[0090] The computing device 510 can be of various types, such as a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Since computers and networks are constantly changing in nature, the description of the computing device 510 shown in FIG. 5 is intended only as a specific example for explaining some embodiments. The computing device 510 can have many other configurations with more or fewer components than the computing device shown in FIG. 5.

[0091] Although several embodiments have been described and illustrated herein, various other means and / or structures may be utilized to implement the functions described herein and / or to obtain the results and / or one or more advantages, and each such variation and / or modification is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be illustrative for example purposes only, and the actual parameters, dimensions, materials, and / or configurations depend on the specific application(s) for which the teachings are used. One of ordinary skill in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments described herein. Thus, the foregoing embodiments are presented only for purposes of example and it is to be understood that embodiments may be practiced otherwise than as specifically described and claimed within the scope of the appended claims and their equivalents. Embodiments of the present disclosure are directed to the individual features, systems, articles, materials, kits, and / or methods described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included in the scope of the present disclosure so long as such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.

[0092] In some embodiments, a method implemented by one or more processors is provided, the method including identifying a source cross-attention map resulting from using a cross-attention layer of a large language image (LLI) model when generating a source image based on processing a source natural language (NL) prompt using the LLI model. The method further includes identifying one or more source random seeds utilized when generating the source image based on processing the source NL prompt using the LLI model. After generating the source image, the method further includes receiving a user interface input indicative of an edit to the source NL prompt used when generating the source image. In response to receiving the user interface input indicative of an edit to the source NL prompt, the method further includes generating an edited image that is visually similar to the source image but that incorporates visual modifications consistent with the edit to the NL prompt indicated by the user interface input in multiple iterations of processing using the LLI model. Generating the edited image in an iteration of processing using the LLI model can include processing one or more features generated based on the edit to the source NL prompt and the source random seeds in the iteration of processing using the LLI model, and injecting at least a portion of the source cross-attention map in at least some of the iterations of generating the edited image using the LLI model.

[0093] These and other embodiments of the technology disclosed herein may include one or more of the following features.

[0094] In some embodiments, the editing includes replacement of a subset of tokens of the source NL prompt with one or more replacement tokens that are different from the subset of tokens of the source NL prompt. In some versions of those embodiments, one or more features generated based on the editing to the source NL prompt include text embeddings of a modified prompt that conforms to the source NL prompt, but replaces a subset of tokens of the source NL prompt with the edited tokens. In some of those versions, in at least some of the iterations of generating the edited image using the LLI model, injecting at least a portion of the source cross-attention map includes using the entire source cross-attention map when processing the text embeddings of the modified prompt in at least some of the iterations. In some variants of those versions, at least some of the iterations are a subset of the iterations, and in other iterations not included in the subset of iterations, other cross-attention maps are utilized when processing the text embeddings, and the source cross-attention map is not utilized when processing the text embeddings. For example, the subset of iterations can be an initial consecutive sequence of iterations, can be more than 5 percent of the iterations, and can include less than 95 percent of the iterations, and / or can be more than 10 percent of the iterations and can include less than 90 percent of the iterations.

[0095] In some embodiments, the editing includes adding one or more additional tokens to the source NL prompt. In some versions of those embodiments, one or more features generated based on the edit to the source NL prompt include text embeddings of the modified prompt that includes the source NL prompt and the additional tokens. In some variations of those versions, in at least some of the iterations of generating the edited image using the LLI model, injecting at least a portion of the source cross-attention map includes using the entire source cross-attention map when processing a portion of the text embedding corresponding to the source NL prompt, and the source cross-attention map is not utilized when processing an additional portion of the text embedding corresponding to the additional tokens. In some of those variations, at least some of the iterations are a subset of the iterations, and in other iterations not included in the subset, the source cross-attention map is not utilized when processing a portion of the text embedding corresponding to the source NL prompt. For example, the subset of iterations can be an initial consecutive sequence of iterations, can be more than 5 percent but less than 95 percent of the iterations, and / or can be more than 20 percent but less than 75 percent of the iterations.

[0096] In some embodiments, the editing includes adjusting the emphasis in one or more emphasized tokens among the source tokens of the source NL prompt, and the adjustment of the emphasis is an increase or decrease in the emphasis. In some versions of those embodiments, one or more features generated based on the editing to the source NL prompt include one or more scaled attention maps for one or more emphasized tokens, and the method includes identifying the emphasized portion of the source cross-attention map corresponding to one or more emphasized tokens, and generating one or more scaled attention maps by scaling the emphasized portion corresponding to the adjustment of the emphasis. In some variations of those versions, the adjustment of the emphasis is an increase in the emphasis, and generating one or more scaled attention maps by scaling the emphasized portion corresponding to the adjustment of the emphasis includes increasing the value of the emphasized portion by a coefficient. In some of those variations, the increase in the emphasis indicated by the user interface input is of a particular magnitude that is one of a plurality of candidate magnitudes, and the coefficient is proportional to the particular magnitude. Optionally, in some embodiments including the editing that is the adjustment of the emphasis in the emphasized token(s), the text embedding of the source NL prompt is processed in an iteration of processing using the LLI model, and the text embedding includes an emphasized embedding portion corresponding to one or more emphasized tokens, and the remaining portion of the embedding corresponding to the remaining portion of the source NL prompt after excluding the emphasized portion.Furthermore, optionally, at least a portion of the source cross-attention map is the remaining portion of the source cross-attention map after excluding the highlighted portion, and in at least some of the iterations of generating the edited image using the LLI model, injecting at least a portion of the source cross-attention map includes using the remaining portion of the source cross-attention map when processing the embedding of the remaining portion in at least some of the iterations, where one or more scaled source cross-attention maps are utilized in at least some of the iterations when processing the highlighted embedding portion.

[0097] In some embodiments, at least some of the iterations are all of the iterations.

[0098] In some embodiments, the cross-attention map includes values that associate tokens of the NL prompt with pixels of the source image. In some of those embodiments, the values respectively define the corresponding weights of the corresponding tokens of the tokens at the corresponding pixels of the pixels.

[0099] In some embodiments, the method further includes generating a source image based on processing a source natural language (NL) prompt using an LLI model.

[0100] In some embodiments, the user interface input indicating an edit to the source NL prompt includes typed input and / or interaction with a graphical user interface that renders the source NL prompt. In some versions of those embodiments, the edit includes adjusting the emphasis in one or more highlighted tokens of the source tokens of the source NL prompt, and the adjustment of the emphasis is an increase or decrease in emphasis. In some of those versions, the user interface input includes interaction with a graphical user interface, and the interaction includes interaction with a slider corresponding to one or more highlighted tokens.

[0101] In some embodiments, the user interface input indicating an edit to the source NL prompt includes spoken input captured in audio data. In some of those embodiments, the method further includes processing the audio data to generate recognized text corresponding to the spoken input using an automatic speech recognition model, and processing the recognized text to determine an edit to the source NL prompt.

[0102] In some embodiments, a method implemented by one or more processors is provided, the method including identifying a real image captured by a real camera and identifying a natural language (NL) caption for the real image. The method further includes generating a noise vector for the real image using an inversion process and based on the real image. The method further includes processing the NL caption using a large language-image (LLI) model and the noise vector to generate a source image that approximates the real image. The method further includes identifying a source cross-attention map resulting from using a cross-attention layer of the LLI model when generating the source image. The method further includes identifying a source random seed utilized when generating the source image. After generating the source image, the method further includes receiving a user interface input indicative of an edit to the NL caption used when generating the source image. In response to receiving the user interface input indicative of an edit to the NL caption, the method further includes generating an edited image including visual modifications that are visually similar to the source image but consistent with the edit to the NL caption indicated by the user interface input in a plurality of iterations of processing using the LLI model. Generating the edited image in a plurality of iterations of processing using the LLI model can include processing one or more features generated based on the edit to the source NL caption and the source random seed in an iteration of processing using the LLI model, and injecting at least a portion of the source cross-attention map in at least some of the iterations of generating the edited image using the LLI model.

[0103] These and other embodiments of the technology disclosed herein may include one or more of the following features.

[0104] In some embodiments, the NL caption for the real image is generated based on other user interface inputs.

[0105] In some embodiments, the NL caption for the real image is automatically generated based on processing the real image using an additional model trained to predict a caption for the image.

[0106] In some embodiments, the inversion process includes using a deterministic denoising diffusion implicit model (DDIM).

[0107] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more tensor processing units (TPUs)) to perform methods such as one or more of the methods described herein. Still other embodiments may include one or more computer systems including one or more processors operable to execute the stored instructions to perform methods such as one or more of the methods described herein.

Claims

**Claim 1** A method implemented by one or more processors, the method comprising: identifying a source cross-attention map resulting from using a cross-attention layer of a large language image (LLI) model when generating a source image based on processing a source natural language (NL) prompt using the LLI model; identifying one or more source random seeds utilized when generating the source image based on processing the source NL prompt using the LLI model; subsequent to generating the source image, receiving a user interface input indicative of an edit to the source NL prompt used when generating the source image; in response to receiving the user interface input indicative of the edit to the source NL prompt, generating, in multiple iterations of processing using the LLI model, an edited image that is visually similar to the source image but includes visual modifications that match the edit to the NL prompt indicated by the user interface input, wherein generating the edited image in the iteration of processing using the LLI model comprises: in the iteration of processing using the LLI model, processing one or more features generated based on the edit to the source NL prompt, and the source random seed; and injecting at least a portion of the source cross-attention map in at least some of the iterations of generating the edited image using the LLI model. A method comprising the above. **Claim 2** The method of claim 1, wherein the edit comprises replacing a subset of tokens of the source NL prompt with one or more replacement tokens different from the subset of tokens of the source NL prompt. **Claim 3** The method of claim 2, wherein the one or more features generated based on the edit to the source NL prompt comprise a text embedding of a modified prompt, the text embedding of the modified prompt conforming to the source NL prompt but replacing the subset of tokens of the source NL prompt with the edited tokens. **Claim 4** In at least some of the iterations of generating the edited image using the LLI model, injecting at least a portion of the source cross-attention map is In at least some of the iterations, when processing the text embedding of the modified prompt, using the entire source cross-attention map, the method according to claim 3, comprising.

5. At least some of the iterations are a subset of the iterations, and in other iterations not included in the subset of the iterations, other cross-attention maps are utilized when processing the text embedding, and the source cross-attention map is not utilized when processing the text embedding, the method according to claim 4.

6. The subset of the iterations is an initial consecutive sequence of the iterations, the method according to claim 5.

7. The subset of the iterations is more than 5 percent of the iterations and includes less than 95 percent of the iterations, the method according to claim 5 or claim 6.

8. The subset of the iterations is more than 10 percent of the iterations and includes less than 90 percent of the iterations, the method according to claim 5 or claim 6.

9. The method according to claim 1, wherein the editing includes adding one or more additional tokens to the source NL prompt.

10. The one or more features generated based on the editing to the source NL prompt include text embeddings of the source NL prompt and a modified prompt including the additional tokens, the method according to claim 9.

11. In at least some of the iterations of generating the edited image using the LLI model, injecting at least a portion of the source cross-attention map is Including using the entire source cross-attention map when processing a portion of the text embedding corresponding to the source NL prompt, and the source cross-attention map is not utilized when processing an additional portion of the text embedding corresponding to the additional tokens, the method according to claim 10.

12. Of the repetitions, at least some are a subset of the repetitions, and in other repetitions not included in the subset, the source cross-attention map is not utilized when processing the portion of the text embedding corresponding to the source NL prompt. The method according to claim 11.

13. The subset of the repetitions is an initial continuous sequence of the repetitions. The method according to claim 12.

14. The subset of the repetitions is more than 5 percent and less than 95 percent of the repetitions. The method according to claim 12 or claim 13.

15. The subset of the repetitions is more than 20 percent and less than 75 percent of the repetitions. The method according to claim 12 or claim 13.

16. The editing includes adjusting the emphasis in one or more emphasized tokens among the source tokens of the source NL prompt, and the adjustment of the emphasis is an increase or decrease in the emphasis. The method according to claim 1.

17. The one or more features generated based on the editing of the source NL prompt include one or more scaled attention maps for the one or more emphasized tokens, and further identifying an emphasized portion of the source cross-attention map corresponding to the one or more emphasized tokens; generating the one or more scaled attention maps by scaling the emphasized portion corresponding to the adjustment of the emphasis. The method according to claim 16.

18. The adjustment of the emphasis is an increase in the emphasis, and generating the one or more scaled attention maps by scaling the emphasized portion corresponding to the adjustment of the emphasis is increasing the value of the emphasized portion by a coefficient. The method according to claim 17.

19. The increase in the emphasis indicated by the user interface input is of a specific magnitude that is one of the magnitudes of a plurality of candidate sizes, and the coefficient is proportional to the specific magnitude. The method according to claim 18.

20. The text embedding of the source NL prompt is processed in the iteration of the processing using the LLI model, and the text embedding includes an emphasized embedding part corresponding to the one or more emphasized tokens and an embedding of the remaining part of the source NL prompt after excluding the emphasized part. The method according to any one of claims 17 to 19.

21. At least a part of the source cross-attention map is the remaining part of the source cross-attention map after excluding the emphasized part, and in at least some of the iterations of generating the edited image using the LLI model, injecting at least a part of the source cross-attention map means In at least some of the iterations, it includes using the remaining part of the source cross-attention map when processing the embedding of the remaining part. The one or more scaled source cross-attention maps are utilized when processing the emphasized embedding part in at least some of the iterations. The method according to claim 20.

22. At least some of the iterations are all of the iterations. The method according to claims 16 to 21.

23. The cross-attention map includes values that associate the tokens of the NL prompt with the pixels of the source image. The method according to any one of claims 1 to 22.

24. Each of the values defines the corresponding weight of the corresponding token of the token at the corresponding pixel of the pixel. The method according to claim 23.

25. Generating the source image based on processing the source natural language (NL) prompt using the LLI model The method according to any one of claims 1 to 24, further comprising.

26. The user interface input indicating the editing to the source NL prompt includes typed input and / or interaction with a graphical user interface that renders the source NL prompt. The method according to any one of claims 1 to 25.

27. The editing includes adjusting the emphasis in one or more emphasized tokens among the source tokens of the source NL prompt, and the adjustment of the emphasis is an increase or decrease in the emphasis, The method according to claim 26, wherein the user interface input includes the interaction with the graphical user interface, and the interaction includes an interaction with a slider corresponding to the one or more emphasized tokens.

28. The user interface input indicating the editing to the source NL prompt includes speech input captured in audio data, and further, processing the audio data to generate recognized text corresponding to the speech input using an automatic speech recognition model; and processing the recognized text to determine the editing to the source NL prompt. The method according to any one of claims 1 to 27.

29. A method implemented by one or more processors, the method comprising: identifying a real image captured by a real camera; identifying a natural language (NL) caption for the real image; generating a noise vector for the real image using an inversion process and based on the real image; processing the NL caption to generate a source image approximating the real image using a large language-image (LLI) model and the noise vector; identifying a source cross-attention map generated using a cross-attention layer of the LLI model when generating the source image; identifying a source random seed utilized when generating the source image; subsequent to generating the source image, receiving a user interface input indicating an editing to the NL caption used when generating the source image; in response to receiving the user interface input indicating the editing to the NL caption, In multiple iterations of the process using the LLI model, generating an edited image that is visually similar to the source image but includes visual modifications consistent with the editing to the NL caption indicated by the user interface input, and in the multiple iterations of the process using the LLI model, generating the edited image comprises in the iteration of the process using the LLI model, processing one or more features generated based on the editing to the source NL caption, and the source random seed, and injecting at least a portion of the source cross-attention map in at least some of the iterations of the generation of the edited image using the LLI model. A method comprising. **Claim 30** The method according to claim 29, wherein the NL caption for the real image is generated based on other user interface inputs. **Claim 31** The method according to claim 29, wherein the NL caption for the real image is automatically generated based on processing the real image using an additional model trained to predict a caption for the image. **Claim 32** The method according to any one of claims 29 to 31, wherein the inversion process comprises using a deterministic denoising diffusion implicit model (DDIM). **Claim 33** A system comprising a memory storing instructions and one or more processors operable to execute the instructions to perform the method according to any one of claims 1 to 32. **Claim 34** One or more temporary or non-temporary computer-readable media storing instructions, which when executed by one or more processors, cause the method according to any one of claims 1 to 32 to be performed.

Images