Hint-to-hint image editing with cross attention control

Abstract

Some implementations relate to editing a source image, wherein the source image is an image generated based on processing source natural language (NL) cues using a large scale language-image (LLI) model. These implementations edit the source image based on user interface input indicating editing of the source NL cue, and optionally independent of any user interface input specifying a mask in the source image and / or independent of any other user interface input. Some implementations of the present disclosure additionally or alternatively relate to applying cue-to-cue editing techniques to edit a source image that is an image generated based on a real image and approximates the real image.

Description

Background Art

[0001] Large-scale language-image (LLI) models such as Google's IMAGEN have demonstrated excellent generative semantics and composition capabilities and have received unprecedented attention from the research community and the public. These LLI models are trained on extremely large language-image datasets and use state-of-the-art image generation models, such as autoregressive and / or diffusion models. These LLI models enable the generation of images conditioned on plain text, known as text-to-image synthesis. For example, these LLI models enable the generation of realistic images reflecting a dog riding on a bicycle in response to a plain text prompt of "photo of dog riding on a bicycle". Various LLI models have recently emerged that demonstrate unprecedented semantic generation.

[0002] Image editing is one of the most fundamental tasks in computer graphics and involves the process of modifying an input image by using auxiliary inputs such as labels, scribbles, masks, or reference images.

[0003] However, many LLI models do not provide easy editing means for generated images, and often lack control over specific semantic regions of a given image (e.g., using only text guidance). For example, even the slightest change in the text prompt may result in a completely different output image being generated using an LLI model. For example, changing "photo of dog riding on a bicycle" to "photo of white dog riding on a bicycle" may produce a completely different generated image, such as one in which the shape of the dog is changed.

[0004] To circumvent this, many proposed LLI-based editing methods require the user to explicitly mask the portion of the image to be inpainted, and drive the edited image to change only in the masked area while matching the background of the original image. However, the masking process is cumbersome (e.g., requiring a lot of user input to define the mask), thus hindering fast and intuitive text-driven editing. In addition, masking the image content removes important structural information, which is completely ignored in the inpainting process. Therefore, some editing capabilities go beyond the scope of inpainting, such as modifying the texture of specific objects.

[0005] A particularly intuitive way to edit images is through textual hints provided by the user. However, previously proposed LLI-based editing methods may lack the ability to edit the generated images through textual hints entirely, or only through textual hints. Summary of the invention

[0006] Some implementations of the present disclosure relate to editing a source image, wherein the source image is an image generated based on processing a source natural language (NL) prompt using a large-scale language-image (LLI) model. These implementations edit the source image based on a user interface input indicating an edit to the source NL prompt, and optionally edit the source image independently of any user interface input that specifies a mask in the source image and / or independently of any other user interface input. More specifically, these implementations generate an edited image that is visually similar to the source image but includes a visual modification consistent with the edit to the source NL prompt. In doing so, various implementations may utilize the same random seed utilized when generating the source image, and may also utilize an internal cross-attention map generated when generating the source image using the LLI model to process the source NL prompt. The cross-attention map is a high-dimensional tensor that binds pixels and tokens extracted from the prompt text. For example, various implementations may inject at least some of the cross-attention maps during at least some iterations of the diffusion process based on the edited prompt, thereby controlling which pixels focus on which tokens of the edited prompt text during which diffusion steps.

[0007] Thus, various implementations provide an intuitive image editing interface by editing only the textual prompts utilized when generating the source image (also referred to herein as prompt-to-prompt editing). This enables voice-based, typed (e.g., physical or virtual keyboards), and / or touch-based (e.g., interaction with emphasized elements, selection of alternative terms) input to edit the source image, and avoids the need for any specification of image masks and / or other input. Such input for editing is natural, can be performed with low latency, and enables various editing tasks that would otherwise be challenging. In addition, the implementations disclosed herein do not require additional and computationally expensive model training, fine-tuning, additional data, or optimization.

[0008] As a non-limiting example, assume that the source NL prompt is "a furry bear watching a bird", the source image reflects a furry bear watching a red bird, and the source image is generated based on processing "a furry bear watching a bird" and a random seed using the LLI model. Editing the source NL prompt may include replacing a subset of the word-grams of the source NL prompt with replacement word-grams (e.g., replacing "bird" with "butterfly"), adding word-grams to the source NL prompt (e.g., adding "blue" before "bird"), and / or adjusting the emphasis of word-grams of the source NL prompt (e.g., increasing the emphasis on "fuzzy").

[0009] An implementation may generate an edited image by processing features generated based on an edit to a source NL cue using an LLI model, a source random seed, and, in at least some of the iterations of the processing, at least a portion of a cross-attention map generated when generating the source image. Utilizing the cross-attention map in conjunction with the source random seed to generate the edited image produces an edited image that is visually similar to the source image but includes visual modifications consistent with the edit. For example, if "bird" is replaced with "butterfly," the edited image may replace "red bird" of the source image with "butterfly," but is otherwise visually very similar. Additionally, for example, if "blue" is added before "bird," the edited image may replace "red bird" with "blue bird," but is otherwise visually very similar. As yet another example, if the emphasis on "furry" is increased, the edited image may replace "bear" with "furrier bear" (e.g., more and / or longer fur), but otherwise be visually very similar. It is worth noting that utilizing the source random seed without utilizing the cross-attention map may produce an image that is visually dissimilar to the source image.

[0010] Some implementations of the present disclosure involve applying the prompt-to-prompt editing techniques disclosed herein to edit a source image, which is an image generated based on a real image and approximates the real image. In these implementations, the edited initial prompt can be, for example, a prompt specified by inputting a user interface and / or a prompt automatically generated (e.g., using an automatic commentary adding model). In addition, in some of these implementations, the source image is generated by generating a noise vector of the real image (e.g., using an inversion process) and processing the initial prompt using the LLI model and the noise vector to generate a source image that approximates the real image.

[0011] It should be understood that all combinations of the aforementioned concepts and additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1ASchematically depicted are example components and interactions that may be involved in generating a source image based on processing a natural language prompt using an LLI model, and generating an edited image using the LLI model that is visually similar to the source image but includes visual modifications consistent with edits to the NL prompt used to generate the source image.

[0013] Figure 1B Schematically depicted are example components and interactions that may be involved in using an LLI model to generate a source image that approximates a real image, and using the LLI model to generate an edited image that is visually similar to the source image but includes visual modifications consistent with edits to the NL cues used to generate the source image.

[0014] Figure 2 An example method is shown for generating source images based on processing natural language cues using an LLI model and storing random seeds used in the processing and cross-attention maps produced in the processing.

[0015] Figure 3 An example method is shown for using an LLI model to generate an edited image that is visually similar to a source image but includes visual modifications consistent with edits to the NL cues used to generate the source image.

[0016] Figure 4 An example method is shown of generating a source image that approximates a real image by processing a natural language hint of a real image and a noise vector of the real image using an LLI model and storing a random seed used in the processing and a cross-attention map generated in the processing.

[0017] Figure 5 An example computer architecture is schematically illustrated upon which selected aspects of the present disclosure may be implemented. DETAILED DESCRIPTION

[0018] Before turning to the drawings, a non-limiting overview of various implementations is provided.

[0019] As a working example of various implementations disclosed herein, let I be a source image generated by an LLI model (e.g., a text-guided diffusion model) using a prompt P and a random seed s. Some implementations attempt to edit the source image I only under the guidance of an edited prompt P*, thereby producing an edited image I*. For example, consider a source image I generated from a prompt P "my new bicycle", and assume that a user wants to edit the color of the bicycle, its material, or even replace it with a scooter while preserving the appearance and structure of the source image I. The intuitive interface for the user is to directly change the text prompt P by further describing the appearance of the bicycle (e.g., adding "green" before "bicycle") or replacing it with another word (e.g., replacing "bicycle" with "scooter"). In contrast to some prior arts, various implementations disclosed herein avoid relying on any user-defined (e.g., defined by interaction with the source image I) mask to assist or indicate where the edits to the source image I should occur. For example, these various implementations avoid relying on any user-defined mask that defines "bicycle" in the source image I and is generated based on user interaction with the source image I. In addition, the various implementations disclosed herein recognize that using the LLI model to process only (a) the same random seed s used in generating the source image (I) and (b) the edited text prompt P* (instead of the original text prompt) produces a completely different image with a different structure and composition. For example, if (b) the edited text prompt P* is "my newgreen bicycle" (where "green" is added before "bicycle"), then processing only (a) the same random seed s and (b) the edited text prompt P* can produce a generated image including "green bicycle". However, relative to the source image I, such a generated image will have a different structure and composition (e.g., will include different background objects).

[0020] The implementations disclosed herein recognize that the structure and appearance of the generated image depends not only on the random seed s, but also on the interaction between pixels and text embeddings through a diffusion process. More specifically, the implementations disclosed herein recognize that by modifying the pixel-to-text interactions that occur in the cross-attention layers, a hint-to-hint image editing capability that maintains the structure and composition of the source image I being edited is achieved. More specifically, injecting at least some of the cross-attention maps generated when generating the source image I when generating the edited image I* using the LLI model enables the composition and structure of the source image I to be preserved.

[0021] For additional context on criss-cross attention maps, a specific example of criss-cross attention in the IMAGEN LLI model (which includes a diffusion model conditioned on text) is described in more detail. The IMAGEN implementation includes three diffusion models conditioned on text: a 64 x 64 model for text generation to images, and two super-resolution models - 64 x 64 256 x 256 model and 256 x 256 1024 x 1024 models. These predict the noise for t ranging from T to 1 via a U-shaped network ,in is the latent vector and c is the text embedding. More specifically, the 64 x 64 model starts with a random noise seed and uses a U-Net. The model is conditioned on the text embedding via both cross-attention layers at downsampled and upsampled resolutions within the U-Net and hybrid attention layers at the resolution. The 256 x 256 model is conditioned on a simply upsampled 64 x 64 image. An efficient version of U-Net is used that includes a hybrid attention layer in the bottleneck (resolution 32). The 1024 x 1024 model was conditioned on a simply upsampled 256 x 256 image. An efficient version of U-Net was used that only included a criss-cross attention layer in the bottleneck (resolution 64).

[0022] With such IMAGEN LLI models and / or other LLI models, the composition and geometry are primarily determined by the resolution of the output of the text-generated image model / input to the initial super-resolution model (e.g., 64 x 64 in the previous example). Therefore, some implementations may perform adaptation using the super-resolution process as is at the text-generated image diffusion process when generating the edited image. When generating images using the IMAGEN LLI model and / or other LLI models, each diffusion step or iteration t includes using a U-shaped network to extract the super-resolution image from the noisy image. and text embedding In the final diffusion step, the process results in the generated image Notably, the interaction between the two modalities occurs during noisy prediction, where embeddings of visual and textual features are fused using a criss-cross attention layer that produces a spatial attention map for each text token.

[0023] More formally, via the learned linear projection , the noisy image The deep spatial features are projected onto the query matrix , and project the text embedding into the key matrix Sum Matrix Then, the attention map is , where unit Define the weight of the value of the jth token at pixel i, and where d is the latent projection dimension of the key and query. Finally, the cross-attention output is defined as , which is then used to update the spatial features .

[0024] Intuitively, the cross-attention output is a weighted average of the values ​​V, where the weights are the attention maps M, which are associated with the similarity between the query matrix Q and the key matrix K. In practice, to increase their expressiveness, multiple attention heads can be used in parallel, and the results are then concatenated and passed through a learned linear layer to get the final output.

[0025] IMAGEN and / or other LLI models condition on textual cues in the noisy predictions of each diffusion step through two types of attention layers: i) crisscross attention layers and ii) hybrid attention, which acts as both self-attention and crisscross attention by concatenating a sequence of text embeddings to a key-value pair of each self-attention layer. Both of them can be called crisscross attention, as various implementations can intervene only in the crisscross attention part of the hybrid attention. That is, only the last channel involving text tokens is modified in the hybrid attention module.

[0026] When controlling the cross-attention in the IMAGEN LLI model and / or other LLI models, it is again noted that the spatial layout and geometry of the generated image depends on the cross-attention map produced when generating the image. This interaction between pixels and text can be observed from plotting the average attention map produced when generating the image. In such plots, it can be observed that pixels are more attracted to the words that describe them. For example, for a prompt including the word "bear", it can be observed that pixels depicting a bear are associated with the word "bear". Such observations indicate that the structure of the image has been determined in the early steps of the diffusion process.

[0027] Since attention reflects the overall composition, when generating an edited image I*, the attention map M obtained from the generation of the source image I using the original prompt P and the LLI model can be injected into the second generation using the modified prompt P* and the LLI model. This allows the synthesis of an edited image I* that is not only manipulated according to the edited prompt, but also preserves the structure of the input image I. Such examples are specific instances of a broader set of attention-based manipulations that implement different types of intuitive editing. Therefore, the following paragraphs describe a more general framework, followed by details of various specific editing operations.

[0028] make is the calculation of a single step t of the diffusion process, which outputs the noise image and attention map (Omit if not used). represents the diffusion step, where the attention map M is attached to the given graph Overwrites, but keeps the value V from the supplied hint. Indicates the use of edited prompts The resulting attention map. Edit ( ) is defined as a universal editing function that receives as input the t-th attention map of the original image and the edited image during their generation.

[0029] A general algorithm for controlled image generation can include performing an iterative diffusion process for two prompts simultaneously, where an attention-based manipulation is applied at each step according to the desired editing task. The internal randomness used in each diffusion process can be fixed / same in each process, which can be reflected by a random seed. This is due to the nature of the diffusion model, where two random seeds produce completely different outputs even for the same prompt. Formally, our general algorithm is:

[0030] More formally, the general algorithm for various implementations can be: -

[0031] It is worth noting that in the aforementioned algorithm, the image I generated by the hint P and the random seed s can be defined as additional inputs. However, the algorithm will remain the same. Also, note that in the aforementioned algorithm, the forward call in line 7 can be skipped by applying the edit function within the diffusion forward function. Additionally or alternatively, the images I generated by the hint P and the random seed s can be defined as additional inputs. However, the algorithm will remain the same. and Both apply a diffusion step and thus have only one step overhead relative to the original inference of the diffusion model.

[0032] Now we provide some examples of specific editing operations that can be used to define Examples of these include word swapping (also called substitution), adding new phrases (also called addition), and attention re-weighting (also called emphasis adjustment).

[0033] Through word exchange, a user interface input is provided, which indicates that the user has exchanged the word-gram of the original prompt for other word-grams. For example, when the user interface input indicates that the original prompt "a big red bicycle" is edited to the edited prompt "a big red car", "bicycle" can be exchanged for "car". Such user interface input can delete "bicycle" and type "car" via touch and / or typed input and / or via verbal user interface input (for example, verbal input "replace bicyclewith car"). Through word exchange and / or other editing operations, the challenge is to retain the original composition while processing the content of the edited prompt. To this end, the implementation injects the attention map generated when generating the source image into the edited image generated using the edited prompt. However, the proposed attention injection may over-constrain geometry, especially when it involves large structural modifications such as "bicycle" to "car". In some implementations of word exchange editing, such over-constraints on geometry can be handled by softer attention constraints. For example, a softer attention constraint can be expressed using the edit function:

[0034] In the preceding editing function, is a timestamp / iteration parameter that determines until which step the injection is applied. Note that the composition is determined in an earlier step of the diffusion process. Therefore, by limiting the number of injection steps, the composition of the newly generated image can be guided while still allowing the geometric freedom necessary to adapt to new cues. An additional or alternative adaptation is to assign different numbers of injection timestamps to different tokens in the cues. In cases where different numbers of tokens are used to represent two words, the graphs can be replicated / averaged as needed using an alignment function such as the one described with respect to adding new phrases.

[0035] By adding a new phrase, a user interface input is provided that indicates that the user has added a new word-meta to the original prompt. For example, such prepending may be performed when the user interface input indicates that "children drawing of" is to be prepended to the original prompt "a castle next to a river". For example, the user interface input may include typing "children drawing of" at the beginning of the original prompt or may be a verbal user interface input such as "prepend children drawing of". By adding the new phrase, in order to retain common details, an implementation may apply attention injection only to common words-meta from the two prompts. For example, attention injection may only be applied to "a castle next to a river" in the aforementioned example. More formally, an alignment function A may be utilized that receives information from the edited prompts. Token index of The corresponding token index in , or None if there is no match. Using this type of alignment function, the example edit function can be expressed as:

[0036] In the aforementioned edit function, recall that index i corresponds to a pixel value, where j corresponds to a text word. Optionally, the aforementioned edit function may optionally and similarly to the word swap edit function set a timestamp to control the number of diffusion steps in which the injection is applied. This type of editing function implements various hint-to-hint capabilities such as stylization, specification of object properties, or global manipulation.

[0037] By attention re-weighting, a user interface input is provided that indicates the degree to which the user desires that the word-grams of the original cue influence the original source image intensify or weaken. For example, the original cue may be "a fluffy red ball", and the user may want an edited image in which the ball is more or less fluffy than it was in the original image. The user interface input indicating an increase or decrease in fluffiness may be, for example, an interaction with a slider or up and down arrows presented with "fluffy", a bold or underlined "fluffy", and / or a verbal input (e.g., "more fluffy"). By attention re-weighting the word-grams of the original cue, the user interface input is provided to indicate the degree to which the user desires that the word-grams of the original cue influence the original source image intensify or weaken. For example, the original cue may be "a fluffy red ball", and the user may want an edited image in which the ball is more or less fluffy than it was in the original image. The user interface input indicating an increase or decrease in fluffiness may be, for example, an interaction with a slider or up and down arrows presented with "fluffy", a bold or underlined "fluffy", and / or a verbal input (e.g., "more fluffy"). To adjust the attention map of the assigned word-gram j*, the word-gram corresponding to the word-gram involved in the emphasized user interface input. For example, when the emphasized input indicates a decrease, the scaling parameter c can be a negative parameter, and when the emphasized input indicates an increase, the scaling parameter c can be a positive parameter, and the scaling parameter c can optionally have a magnitude based on the degree of increase or decrease indicated by the emphasized input. For example, the scaling parameter can be expressed as ∈ [−2, 2]. The rest of the attention map can remain unchanged. Such an editing function can be expressed as

[0038] Some non-limiting examples of practical applications of various implementations are now provided, which demonstrate intuitive plain text editing by controlling the spatial layout corresponding to each word in a user-provided prompt.

[0039] One practical application is to locally edit a source image by editing a user-provided source hint and without the need for any user-provided mask. For example, a source image can be generated using the hint "lemon cake" and the LLI model. The user interface input can replace "lemon" with "pumpkin", resulting in an edited hint "pumpkin cake". By utilizing the implementation disclosed herein, an edited image can be generated that maintains the spatial layout, geometry, and semantics of the source image. On the other hand, simply feeding the hint "pumpkin cake" to the synthesis model produces completely different geometries even when the same random seed is used in a deterministic setting.

[0040] In addition to or in lieu of modifying only the texture, another practical application is to perform structural modifications on the source image. For example, a source image can be generated using a prompt including "bicycle" (among other words) and an LLI model, and a user interface input can replace "bicycle" with "car". By utilizing the implementation disclosed herein, an edited image can be generated that changes the "bicycle" of the source image to a "car" in the edited image. It is observed that the more diffusion steps in which cross-attention injection is applied when generating the edited image, the higher the fidelity to the original image. However, the best results are not necessarily achieved by applying injection in all diffusion steps. Therefore, cross-attention injection can optionally be applied only to a subset of threshold percentages of steps or iterations such as between 5% and 95%, between 15% and 90%, or other limits. Optionally, an interactive user interface element can be presented along with the edited prompt, which enables user input to define the fidelity to the original image that should be followed when generating the edited image. When such user interface elements are provided, the subset of steps or iterations to which cross-attention injection is applied may correspond to user interface input (if any) involving these interactive user interface elements. For example, the interactive user interface element may include a slider, and the number of iterations to which cross-attention injection is applied may be based on the positioning of the slider.

[0041] Another practical application is that a user may wish to add a new designation to a generated source image, rather than replacing one word with another. For example, a generated source image may be generated based on a source prompt "a car on the side of the street", and a user interface input may be provided to add "crushed" in front of the car, thereby generating an edited prompt "a crushed car on the side of the street". In such a case, the attention map of the source prompt may be utilized when generating the edited image, while also allowing the newly added word ("crushed") and the corresponding attention map to be utilized when generating the edited image. This may produce an edited image that includes a crushed car (whereas the source image does not include a crushed car), while still retaining the background of the source image.

[0042] Another practical application is to preserve the image composition of the source image when performing global edits. In such applications, the edit should affect all parts of the image, but still maintain the original composition, such as the position and identity of the objects. For example, editing the source prompt "a car on the side of the street" to "a car in the snowy street" can keep the background and car of the source image while adding snow to the background and car. As another example, editing the source prompt "photo of a waterfall" to "impressionism painting of a waterfall" can keep the original composition of the source image while changing it from a photo to an impressionist painting.

[0043] Although various implementations are described herein regarding applying a prompt-to-prompt editing technique to a source image, which is a source image generated by processing source prompts using an LLI model, implementations of the present disclosure additionally or alternatively involve applying the prompt-to-prompt editing technique disclosed herein to edit a source image, which is a source image generated based on a real image (e.g., captured by a real-world physical camera) and approximates the real image. In these implementations, the edited initial prompt can be, for example, a prompt specified by inputting a user interface and / or a prompt automatically generated (e.g., using an automatic commentary addition model). In addition, in some of these implementations, the source image is generated by generating a noise vector for the real image (e.g., using an inversion process) and processing the initial prompt using the LLI model and the noise vector to generate a source image that approximates the real image.

[0044] The implementations that apply the hint-to-hint editing technique to edit the source image recognize that editing a real image may require finding an initial noise vector that produces a given input image when fed to a diffusion process. This process is often called inversion, but is not routinely used for LLI such as text-guided diffusion models. A simple approach is to add Gaussian noise to the real image and then perform a predefined number of diffusion steps. However, such an approach may produce significant distortion. Therefore, some implementations disclosed herein adopt an improved inversion approach that is based on a deterministic denoising diffusion implicit model (DDIM) model rather than a denoising diffusion probabilistic model (DDPM). These implementations can perform the diffusion process in reverse, i.e. Rather than ,in is set to the real image.

[0045] Such an inversion process can produce satisfactory results. However, in many other cases, such inversion is not accurate enough. This may be due in part to the distortion-editability trade-off, where reducing the classifier-free guidance parameter (i.e., reducing the hint effect) improves reconstruction but constrains the ability to perform significant manipulations. To alleviate this limitation, some implementations use masks extracted directly from the attention map to restore the unedited areas of the original image. Note that in these implementations, the masks are generated without guidance from the user. In addition, in some of these implementations, the approach works even with a simple DDPM inversion scheme (adding noise followed by denoising).

[0046] DDPM is a generative latent variable model that aims to approximate the data distribution. and a distribution that is easy to sample from DDPM The “forward process” from data to noise is modeled in the space of

[0047] The process is from A Markov chain is started where noise is gradually added to the data to generate latent variables . Therefore, the sequence of latent variables follows , where the steps in the forward process are defined as those scheduled by Parameterized Gaussian transfer . When T is large enough, the final noise vector Almost follows an isotropic Gaussian distribution.

[0048] An interesting property of the forward pass is that the latent variable can be transformed without sampling the intermediate latent vectors. Directly expressed as noise and The following linear combination of : ,in .

[0049] In order to distribute Sampling in the posterior Sampling is performed to define isotropic Gaussian noise The dual “inverse process” to the data Due to the intractable reverse process Depends on unknown data distribution , we can use a parameterized Gaussian transfer network to approximate the inverse process. The prediction can be added to Equation 2 using Noise To

[17] replace .

[0050] Under this definition, Bayes' theorem can be used to approximate .

[0051] Once you have trained , you can use the following sampling method: . Each sampling stage can be controlled , and in DDIM, can be used in all steps To make the sampling process deterministic. Finally, the inverse process can be trained by solving the following optimization problem: The parameters are taught by maximizing the variational lower bound To fit .

[0052] Now turning to the attached figure, Figure 1A Schematically depicted are example components and interactions that may be involved in generating a source image 103A based on processing a natural language prompt 101A using an LLI model 150, and generating an edited image 107A using the LLI model 150 that is visually similar to the source image 103A but includes visual modifications consistent with edits to the NL prompt used to generate the source image 103A, which edits are reflected in the prompt edit input 106A.

[0053] exist Figure 1A , the client device 110 may provide a NL prompt 101A, such as a working example of "photo of an orange catriding on a bicycle". The NL prompt 101A may be generated based on a user interface input, such as a typed input or a spoken input, provided by a user at the client device 110. For example, the NL prompt 101A may be based on text from speech recognition performed based on the spoken input received at the client device 110.

[0054] The source image engine 120 may process the NL hint 101A using the LLI model 150 to generate the source image 103A. In generating the source image 103A, one or more random (truly random or pseudo-random) seeds 104A may be utilized. In addition, a cross-attention map 105A is generated in generating the source image 103A. The random seeds 104A and the cross-attention map 105A may be provided to the edited image engine 130.

[0055] The edited image engine 130 receives a prompted edit input 106A, which is a user interface input provided at the client device 110 and specifies one or more edits to the NL prompt 101A, such as a replacement input (e.g., replacing "bicycle" with "horse"), an addition input (e.g., adding "green" before "bicycle"), and / or an emphasis adjustment input (e.g., increasing the emphasis on "orange"). In response to receiving the prompted edit input 106A, the edited image engine 130 may interact with the LLI model 150 in generating an edited image 107A that is visually similar to the source image 103A, but includes visual modifications consistent with the edits to the NL prompt 101A that are reflected by the prompted edit input 106A.

[0056] When interacting with the LLI model 150 in generating the edited image 107A, the edited image engine 130 may utilize the random seed 104A utilized in generating the source image 103A, may utilize edit features based on the edits reflected by the hint edit input 106A (e.g., text embeddings of modified hints reflected by the edits), and may utilize at least some of the cross-attention maps 105A in at least some of the iterations of generating the edited image 107A. Which cross-attention maps 105A are utilized in generating the edited image 107A and / or in which iterations the cross-attention maps are utilized may depend on the type of edit reflected by the hint edit input 106A (e.g., depending on whether the edit is of a replacement, addition, or emphasis adjustment type).

[0057] Figure 1B Schematically depicted are example components and interactions that may be involved in using an LLI model 150 to generate a source image 103B that approximates a real image 102B, and using the LLI model 150 to generate an edited image that is visually similar to the source image 103B but includes visual modifications consistent with edits to the NL prompt used to generate the source image 103B, which are reflected in the prompt edit input 106B.

[0058] exist Figure 1BIn the example embodiment, the client device 110 may provide a real image 102B to the noise vector engine 120. The noise vector engine 120 may generate a noise vector 102B1 for the real image 102B. For example, the noise vector engine 120 may generate the noise vector 102B1 using an inversion process and the real image, such as by using a DDIM or DDPM inversion process. The noise vector 102B1 and the NL hint 101B of the real image 102B are provided to the source image engine 120. The NL hint 101B may be provided by the client device and based on a user interface input (e.g., the user interface input is a user-curated caption of the real image 102B) and / or may be provided by the caption engine 140, which automatically generates the NL hint by processing the real image 102B using a caption model.

[0059] The source image engine 120 can use the LLI model 150 to process the NL hint 101B and the noise vector 102B1 to generate a source image 103B that approximates the real image 102B. When generating the source image 103B, one or more random (truly random or pseudo-random) seeds 104B can be used. In addition, a cross-attention map 105B is generated when generating the source image 103B. The random seeds 104B and the cross-attention map 105B can be provided to the edited image engine 130.

[0060] The edited image engine 130 receives a prompted edit input 106B, which is a user interface input provided at the client device 110 and specifies one or more edits to the NL prompt 101B (which may be rendered at the client device 110 - optionally based on output from the transcript engine 140), such as a replacement input, an addition input, and / or an emphasis adjustment input. In response to receiving the prompted edit input 106B, the edited image engine 130 may interact with the LLI model 150 in generating an edited image 107B that is visually similar to the source image 103B, but includes visual modifications consistent with the edits to the NL prompt 101B that are reflected by the prompted edit input 106B.

[0061] When interacting with the LLI model 150 in generating the edited image 107B, the edited image engine 130 may utilize the random seed 104B utilized in generating the source image 103B, may utilize edit features based on the edits reflected by the prompted edit input 106B (e.g., text embeddings of modified hints reflected by the edits), and may utilize at least some of the cross-attention maps 105B in at least some of the iterations of generating the edited image 107B. Which cross-attention maps 105B are utilized in generating the edited image 107B and / or in which iterations the cross-attention maps are utilized may depend on the type of edit reflected by the prompted edit input 106B (e.g., depending on whether the edit is of a replacement, addition, or emphasis adjustment type).

[0062] Figure 2 An example method 200 is shown for generating a source image based on processing a natural language prompt using an LLI model and storing a random seed used in the processing and a cross-attention map generated in the processing. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computer systems, such as one or more components of a server computing device. In addition, although the operations of method 200 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0063] At block 202, the system receives a natural language prompt. For example, the natural language prompt may be a prompt provided based on a user interface input at a client device, such as a user interface input involving an interface or front end of the system that is accessible via the client device.

[0064] At block 204, the system generates one or more source random seeds. For example, the system can generate the source random seeds using a random or pseudo-random process.

[0065] At block 206, the system generates a source image based on processing the source random seed of block 204 and the NL hint of block 202 using the LLI model. When generating the source image based on processing using the LLI model, a cross-attention map is generated as described herein. The cross-attention map may include values ​​that bind word-grams of the NL hint to pixels of the generated source image.

[0066] At block 208 , the system stores (eg, at least temporarily in memory) the random seed of block 204 and the cross-attention map generated during generation of the source image at block 206 .

[0067] At block 210, the system causes rendering of the source image and the NL prompt 210. For example, the system may cause such rendering to occur at a client device that provided the natural language prompt of block 202.

[0068] Figure 3 An example method 300 is shown for generating an edited image using an LLI model that is visually similar to a source image but includes visual modifications consistent with edits to NL cues used to generate the source image. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computer systems, such as one or more components of a server computing device. Furthermore, while the operations of method 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0069] At block 302, the system receives a user interface input indicating an edit to a source NL hint used in generating a source image. The user interface input may be received at a client device in response to rendering a source image, and optionally in response to rendering the NL hint used in generating the source image. The source image may be Figure 2 The source image of block 206 of iteration of method 200, and the NL hint may be Figure 2 NL hint of block 206 of iteration of method 200. Alternatively, the source image may be Figure 4 The source image of block 410 of the iteration of method 400 (approximately the real image), and the NL hint can be Figure 4 The NL prompt of block 404 of iteration of method 400 .

[0070] In various implementations, box 302 includes one or more of sub-boxes 302A, 302B, and 302C. At sub-box 302A, the user interface of box 302 input includes a replacement input. The replacement input can reflect an edit, which is replacing a subset of the word-grams of the source NL prompt with one or more replacement word-grams that are different from the subset of the word-grams of the source NL prompt. At sub-box 302B, the user interface of box 302 input includes an addition input. The addition input can reflect an edit, which is adding one or more additional word-grams to the source NL prompt. At sub-box 302A, the user interface of box 302 input includes an emphasis adjustment input. The emphasis adjustment input can reflect an edit, which is an emphasis adjustment of one or more emphasized word-grams of the source NL prompt, wherein the adjustment is an increase or decrease in emphasis, and can optionally reflect the magnitude of the increase or decrease.

[0071] At box 304, the system generates an edit feature based on an edit to the source NL prompt, the edit reflected by the user interface input received at box 302. For example, where the edit is a replacement, the system can generate an edit feature that includes a text embedding of the modified prompt, which conforms to the source NL prompt but replaces a subset of the word-grams of the source NL prompt with the edited word-grams. As another example, where the edit is an addition, the system can generate an edit feature that includes a text embedding of the modified prompt, which includes the source NL prompt and the additional word-grams. As another example, where the edit is an emphasis adjustment of an emphasized word-gram, the system can generate an edit feature that includes a scaled attention map of one or more emphasized word-grams.

[0072] At block 306, the system identifies the source seeds and cross-attention maps used in generating the source image of block 302. The source seeds and cross-attention maps may be Figure 2 The method of generating a seed and a cross-attention map of the source image in iteration 200 of the method, or Figure 4 In the iterations of method 400, seed and cross-attention maps of the source image (which approximates the real image) are generated.

[0073] At box 308, the system generates an edited image based on using the LLI model to process: (A) edit features (generated at box 304) generated based on edits to the source NL cues, (B) source seeds (identified at box 306), and (C) at least some of the cross-attention maps (identified at box 306).

[0074] In some implementations, box 308 includes a sub-box 308A in which the system uses only a subset of the cross-attention maps and / or uses the cross-attention maps only in a subset of the processed iterations. In some versions of these implementations, whether to utilize the cross-attention maps or which subset of the cross-attention maps to utilize may depend on an edit to the source NL prompt, which is reflected by the user interface input received at box 302. In addition, in some of these versions or in other versions of these implementations, whether to apply the cross-attention maps only in a subset of the iterations and / or in which subset of the cross-attention maps to utilize may depend on an edit to the source NL prompt, which is reflected by the user interface input received at box 302. For example, in the case where the edit is a replacement, only a subset of the cross-attention maps may be utilized, and only in a subset of the iterations, which subset of the cross-attention maps excludes the cross-attention maps corresponding to the replaced word-gram. As another example, in the case where the edit is an addition, the cross-attention maps may optionally include all cross-attention maps, but they may only be utilized in a subset of the iterations (e.g., not utilized when processing features corresponding to the replaced word-gram). As yet another example, where the edit is an emphasis adjustment to an emphasized word-gram, a first subset of cross-attention maps may be utilized for unemphasized word-grams, and a scaled version of a second subset of cross-attention maps may be utilized for emphasized word-grams.

[0075] In some additional or alternative implementations, sub-box 308A may include the system always using the cross-attention map only in a subset of the iterations processed, such as using the cross-attention map only in a threshold percentage of the iterations. For example, the threshold may be between 5% and 95%, between 15% and 90%, between 25% and 75%, or between other limits. Optionally, in some versions of these additional or alternative implementations, an interactive user interface element may be presented that enables user input to define the fidelity to the original image that should be followed when generating the edited image. In some of these versions, the threshold may be determined by the system based on interaction with the user interface element.

[0076] At block 310 , the system causes rendering of the edited image and optionally the edited NL prompt. For example, the system may cause such rendering at a client device that provided the user interface input of block 302 .

[0077] At optional block 312, the system may monitor for new user interface input that indicates further edits to the source NL prompt that are in addition to the edits of the previous iteration of block 302. If such new user interface input is detected, the system may continue to perform another iteration of blocks 302, 304, 306, 308, and 310 based on such new user interface input.

[0078] Figure 4 An example method 400 is shown for generating a source image that approximates a real image by processing a natural language prompt of a real image and a noise vector of the real image using an LLI model and storing a random seed used in the process and a cross-attention map generated in the process. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computer systems, such as one or more components of a server computing device. In addition, although the operations of method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0079] At block 402 , the system identifies a real image captured by a real camera, such as a real image uploaded from a client device.

[0080] At block 404, the system identifies NL cues for the real image. Upon identifying NL cues for the real image, the system may perform sub-block 404A or sub-block 404B.

[0081] At subblock 404A, an NL prompt for the real image is generated based on the user interface input. For example, when the real image is received from the client device at block 402, an NL prompt may also be received, and the NL prompt may be responsive to the user interface input received at the client device. For example, the user interface input may be received at the client device in response to a rendering prompt, such as a prompt that reads “please provide a natural language description of this image”.

[0082] At sub-block 404B, NL hints for the real image are generated based on processing the real image using a captioning model or other visual language model.

[0083] At block 406, the system generates a noise vector for the real image. For example, the system can generate the noise vector based on applying an inversion process, such as a DDIM or DDPM inversion process, to the real image.

[0084] At block 408, the system generates one or more source random seeds. For example, the system can generate the source random seeds using a random or pseudo-random process.

[0085] At block 410, the system generates a source image that approximates the real image by processing the source random seed of block 408, the noise vector of block 406, and the NL hint of block 404 using the LLI model. When the source image is generated based on such processing using the LLI model, a cross-attention map is generated as described herein. The cross-attention map may include values ​​that bind word-grams of the NL hint to pixels of the generated source image.

[0086] At block 412 , the system stores (eg, at least temporarily in memory) the random seed of block 408 and the cross-attention map generated during generation of the source image at block 410 .

[0087] At optional block 414, the system causes rendering of the source image and / or NL hint 210. For example, the system may cause such rendering to occur at a client device that provided the real image of block 402.

[0088] Figure 5 is a block diagram of an example computing device 510 that may optionally be utilized to perform one or more aspects of the techniques described herein. For example, all or some aspects of computing device 510 may be incorporated into a server or other computing device utilized to implement the hint-to-hint editing techniques disclosed herein.

[0089] The computing device 510 typically includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 including, for example, a memory subsystem 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow interaction with a user of the computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0090] The user interface input device 522 may include a keyboard, a pointing device such as a mouse, a trackball, a touch pad or a graphics tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 510 or onto a communication network.

[0091] The user interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Generally, the use of the term "output device" is intended to include devices and methods for outputting information from the computing device 510 to all possible types of users or another machine or computing device.

[0092] The storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include a computer program that executes Figure 2 , Figure 3 and / or Figure 4 Selected aspects of the methods and logic for implementing the various components described herein.

[0093] These software modules are typically executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 may include multiple memories including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations may be stored by the file storage subsystem 526 in the storage subsystem 524, or in other machines accessible to the processor 514.

[0094] Bus subsystem 512 provides a mechanism for the various components and subsystems of computing device 510 to communicate with each other as intended. Although bus subsystem 512 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0095] Computing device 510 can be of different types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Figure 5 The description of computing device 510 depicted in FIG. 5 is intended only as a specific example for the purpose of illustrating some implementations. Many other configurations of computing device 510 are possible, and these configurations are similar to those of FIG. Figure 5 Computing devices are depicted as having more or fewer components than those shown.

[0096] Although several implementations have been described and illustrated herein, a variety of other means and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which this teaching is used. Those skilled in the art will recognize or confirm many equivalents of the implementations described herein using only routine experiments. Therefore, it should be understood that the aforementioned implementations are presented only by way of example, and it should be understood that within the scope of the attached claims and their equivalents, implementations may be practiced in a manner other than specifically described and claimed. The implementations disclosed herein relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure if such features, systems, articles, materials, kits, and / or methods do not contradict each other.

[0097] In some implementations, a method implemented by a processor is provided, and the method includes: identifying a source cross-attention map generated by a cross-attention layer of a large-scale language-image (LLI) model when generating a source image based on processing a source natural language (NL) prompt using the LLI model. The method also includes: identifying one or more source random seeds utilized when generating the source image based on processing the source NL prompt using the LLI model. The method also includes, after generating the source image, receiving a user interface input indicating an edit to the source NL prompt used when generating the source image. The method also includes, in response to receiving the user interface input indicating the edit to the source NL prompt, generating an edited image in multiple iterations of processing using the LLI model, the edited image being visually similar to the source image but including a visual modification consistent with the edit to the NL prompt indicated by the user interface input. Generating the edited image in the iterations of processing using the LLI model may include: in the iterations of processing using the LLI model, processing: one or more features generated based on the edit to the source NL prompt, and the source random seed; and in at least some of the iterations of generating the edited image using the LLI model, injecting at least a portion of the source cross-attention map.

[0098] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0099] In some implementations, the editing includes replacing a subset of word-grams of the source NL prompt with one or more replacement word-grams that are different from the subset of word-grams of the source NL prompt. In some versions of these implementations, one or more features generated based on the editing of the source NL prompt include a text embedding of the modified prompt, which modified prompt conforms to the source NL prompt but replaces a subset of word-grams of the source NL prompt with the edited word-grams. In some of these versions, injecting at least a portion of the source cross-attention map in at least some of the iterations of generating the edited image using the LLI model includes: in at least some of the iterations, using the entirety of the source cross-attention map when processing the text embedding of the modified prompt. In some variants of these versions, at least some of the iterations are a subset of the iterations, and in other iterations that are not included in the subset of the iterations, other cross-attention maps are utilized when processing the text embeddings, and the source cross-attention map is not utilized when processing the text embeddings. For example, a subset of iterations may: be the initial contiguous sequence of iterations; include more than 5 percent of the iterations but less than 95 percent of the iterations; and / or include more than 10 percent of the iterations but less than 90 percent of the iterations.

[0100] In some implementations, the editing includes adding one or more additional tokens to the source NL prompt. In some versions of these implementations, one or more features generated based on the editing of the source NL prompt include a text embedding of the modified prompt, the modified prompt including the source NL prompt and the additional tokens. In some variants of these versions, injecting at least a portion of the source cross-attention map in at least some of the iterations of generating the edited image using the LLI model includes: using the entirety of the source cross-attention map when processing the portion of the text embedding corresponding to the source NL prompt, wherein the source cross-attention map is not utilized when processing the additional portion of the text embedding corresponding to the additional token. In some of these variants, at least some of the iterations are a subset of the iterations, and in other iterations not included in the subset, the source cross-attention map is not utilized when processing the portion of the text embedding corresponding to the source NL prompt. For example, the subset of iterations can be: an initial continuous sequence of iterations; more than five percent of the iterations, but less than ninety-five percent of the iterations; and / or more than twenty percent of the iterations, but less than seventy-five percent of the iterations.

[0101] In some implementations, the edit includes an emphasis adjustment of one or more emphasized word-lemmas in the source word-lemmas of the source NL prompt, the emphasis adjustment being an emphasis increase or decrease. In some versions of these implementations, one or more features generated based on the edit of the source NL prompt include one or more scaled attention maps for the one or more emphasized word-lemmas, and the method further includes: identifying an emphasized portion of the source cross-attention map corresponding to the one or more emphasized word-lemmas; and generating one or more scaled attention maps by scaling the emphasized portion corresponding to the emphasis adjustment. In some variants of these versions, the emphasis adjustment is an emphasis increase, and generating one or more scaled attention maps by scaling the emphasized portion corresponding to the emphasis adjustment includes increasing the value of the emphasized portion by a factor. In some of these variants, the emphasis increase indicated by the user interface input has a specific magnitude, the specific magnitude being one of a plurality of candidate magnitudes, and wherein the factor is proportional to the specific magnitude. Optionally, in some of the implementations of editing that includes emphasis adjustment for emphasized tokens, a text embedding of a source NL prompt is processed in an iteration of processing using an LLI model, and the text embedding includes an emphasized embedding portion corresponding to one or more emphasized tokens and a rest embedding corresponding to a rest of the source NL prompt after excluding the emphasized portion. Further, optionally, at least a portion of the source cross-attention map is a rest of the source cross-attention map after excluding the emphasized portion, and injecting at least a portion of the source cross-attention map in at least some of the iterations of generating the edited image using the LLI model includes: in at least some of the iterations, using the rest of the source cross-attention map when processing the rest embedding, wherein, in at least some of the iterations, utilizing one or more scaled source cross-attention maps when processing the emphasized embedding portion.

[0102] In some implementations, at least some of the iterations are all of the iterations.

[0103] In some implementations, the cross-attention map includes values ​​that bind word-grams of the NL hint to pixels of the source image. In some of these implementations, the values ​​each define a corresponding weight of a corresponding word-gram in the word-grams on a corresponding pixel in the pixels.

[0104] In some implementations, the method also includes generating a source image based on processing a source natural language (NL) prompt using the LLI model.

[0105] In some implementations, the user interface input indicating the edit of the source NL prompt includes typed input and / or interaction with a graphical user interface that renders the source NL prompt. In some versions of these implementations, the edit includes an emphasis adjustment of one or more emphasized word-grams in the source word-grams of the source NL prompt, the emphasis adjustment being an increase or decrease in emphasis. In some of these versions, the user interface input includes interaction with a graphical user interface, and the interaction includes interaction with a slider corresponding to the one or more emphasized word-grams.

[0106] In some implementations, the user interface input indicating the edit to the source NL prompt comprises a spoken input captured in audio data. In some of these implementations, the method further comprises: processing the audio data using an automatic speech recognition model to generate recognized text corresponding to the spoken input; and processing the recognized text to determine the edit to the source NL prompt.

[0107] In some implementations, a method implemented by a processor is provided, and the method includes identifying a real image captured by a real camera and identifying a natural language (NL) commentary of the real image. The method also includes using an inversion process and generating a noise vector of the real image based on the real image. The method also includes using a large-scale language-image (LLI) model and a noise vector to process the NL commentary to generate a source image that is similar to the real image. The method also includes identifying a source cross-attention map generated by a cross-attention layer of the LLI model when generating the source image. The method also includes identifying a source random seed used when generating the source image. The method also includes, after generating the source image, receiving a user interface input indicating an edit of the NL commentary used when generating the source image. The method also includes, in response to receiving a user interface input indicating an edit of the NL commentary, generating an edited image in multiple iterations of processing using the LLI model, the edited image being visually similar to the source image, but including a visual modification consistent with the edit of the NL commentary indicated by the user interface input. Generating an edited image in multiple iterations of processing using the LLI model may include: in the iterations of processing using the LLI model, processing: one or more features generated based on edits to the source NL commentary, and a source random seed; and in at least some of the iterations of generating the edited image using the LLI model, injecting at least a portion of the source cross-attention map.

[0108] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0109] In some implementations, the NL caption for the real image is generated based on other user interface inputs.

[0110] In some implementations, the NL caption for the real image is automatically generated based on processing the real image using an additional model trained to predict the caption for the image.

[0111] In some implementations, the inversion process includes using a deterministic denoising diffusion implicit model (DDIM).

[0112] Other implementations may include a non-transitory computer-readable storage medium storing instructions that are executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform a method, such as one or more of the methods described herein. Still other implementations may include a system of one or more computers including one or more processors operable to execute the stored instructions to perform a method, such as one or more of the methods described herein.

Claims

1. A method implemented by one or more processors, the method comprising: identifying a source criss-cross attention map produced using a criss-cross attention layer of a large-scale language-to-image (LLI) model when generating a source image based on processing a source natural language (NL) prompt using the LLI model; identifying one or more source random seeds utilized in generating the source image based on processing the source NL cues using the LLI model; After generating the source image: receiving a user interface input indicating an edit to the source NL hint used in generating the source image; In response to receiving the user interface input indicating the editing of the source NL prompt: generating, in a plurality of iterations of processing using the LLI model, an edited image that is visually similar to the source image but includes visual modifications consistent with the edits to the NL cues indicated by the user interface input, wherein generating, in the iterations of processing using the LLI model, the edited image comprises: In the iteration of processing using the LLI model, the process: one or more features generated based on said editing of said source NL prompt, and the source random seed, and At least a portion of the source cross-attention map is injected in at least some of the iterations of generating the edited image using the LLI model.

2. The method of claim 1, wherein: The editing comprises replacing a subset of word-grams of the source NL prompt with one or more replacement word-grams that are different from the subset of word-grams of the source NL prompt.

3. The method of claim 2, wherein: The one or more features generated based on the editing of the source NL prompt include a text embedding of a modified prompt that conforms to the source NL prompt but replaces the subset of word-grams of the source NL prompt with the edited word-grams.

4. The method of claim 3, wherein: Injecting at least a portion of the source cross-attention map in at least some of the iterations of generating the edited image using the LLI model comprises: In said at least some of said iterations, all of said source cross-attention maps are used in processing said text embedding of said modified prompt.

5. The method of claim 4, wherein: At least some of the iterations are a subset of the iterations, and wherein, in other iterations not included in the subset of the iterations, other cross-attention maps are utilized when processing the text embeddings, and the source cross-attention map is not utilized when processing the text embeddings.

6. The method of claim 5, wherein: The subset of the iterations is an initial contiguous sequence of the iterations.

7. The method of claim 5 or claim 6, wherein: The subset of the iterations includes more than five percent of the iterations but less than ninety-five percent of the iterations.

8. The method of claim 5 or claim 6, wherein: The subset of the iterations includes more than ten percent of the iterations but less than ninety percent of the iterations.

9. The method of claim 1, wherein: The editing includes adding one or more additional word-grams to the source NL prompt.

10. The method of claim 9, wherein: The one or more features generated based on the editing of the source NL prompt include a text embedding of a modified prompt, the modified prompt including the source NL prompt and the additional word-gram.

11. The method of claim 10, wherein: Injecting at least a portion of the source cross-attention map in at least some of the iterations of generating the edited image using the LLI model comprises: The entirety of the source cross-attention map is used when processing a portion of the text embedding corresponding to the source NL hint, wherein the source cross-attention map is not utilized when processing an additional portion of the text embedding corresponding to the additional word-gram.

12. The method of claim 11, wherein: At least some of the iterations are a subset of the iterations, and wherein, in other iterations not included in the subset, the source cross-attention map is not utilized when processing the portion of the text embedding corresponding to the source NL hint.

13. The method of claim 12, wherein: The subset of the iterations is an initial contiguous sequence of the iterations.

14. A method according to claim 12 or claim 13, wherein: The subset of the iterations includes more than five percent of the iterations but less than ninety-five percent of the iterations.

15. The method of claim 12 or claim 13, wherein: The subset of the iterations includes more than twenty percent of the iterations but less than seventy-five percent of the iterations.

16. The method of claim 1, wherein: The editing includes adjusting the emphasis of one or more emphasized word-grams in the source word-grams of the source NL prompt, wherein the emphasis adjustment is an increase or decrease in emphasis.

17. The method of claim 16, wherein: The one or more features generated based on the editing of the source NL prompt include one or more scaled attention maps for the one or more emphasized word-grams, and the method further includes: identifying an emphasized portion of the source cross-attention map corresponding to the one or more emphasized word-grams; and The one or more scaled attention maps are generated by scaling the emphasized portion corresponding to the emphasis adjustment.

18. The method of claim 17, wherein: The emphasis adjustment is an emphasis increase, and wherein generating the one or more scaled attention maps by scaling the emphasis portion corresponding to the emphasis adjustment comprises: Increase the value of the emphasized portion by a factor.

19. The method of claim 18, wherein: The increase in emphasis indicated by the user interface input has a particular magnitude, the particular magnitude being one of a plurality of candidate magnitudes, and wherein the factor is proportional to the particular magnitude.

20. The method of any one of claims 17 to 19, wherein: The text embedding of the source NL prompt is processed in the iteration of processing using the LLI model, and wherein the text embedding includes an emphasized embedding portion corresponding to the one or more emphasized word-grams and a remaining embedding corresponding to a remaining portion of the source NL prompt after excluding the emphasized portion.

21. The method of claim 20, wherein: The at least a portion of the source cross-attention map is a remaining portion of the source cross-attention map after excluding the emphasized portion, and wherein injecting the at least a portion of the source cross-attention map in at least some of the iterations of generating the edited image using the LLI model comprises: In at least some of the iterations, the remaining portion of the source cross-attention map is used when processing the remaining portion embedding, wherein in at least some of the iterations, the one or more scaled source cross-attention maps are utilized when processing the emphasized embedding portion.

22. The method of claims 16 to 21, wherein: Said at least some of said iterations are all said iterations.

23. A method as claimed in any preceding claim, wherein: The criss-cross attention map includes values ​​that bind word-grams of the NL hint to pixels of the source image.

24. The method of claim 23, wherein: The values ​​each define a corresponding weight of a corresponding one of the word-grams on a corresponding one of the pixels.

25. The method of any preceding claim, further comprising: The source image is generated based on processing the source natural language NL prompt using the LLI model.

26. A method as claimed in any preceding claim, wherein: The user interface input indicating the edit of the source NL hint comprises typed input and / or interaction with a graphical user interface rendering the source NL hint.

27. The method of claim 26, in, The editing comprises adjusting the emphasis of one or more emphasized word-grams in the source word-grams of the source NL prompt, the emphasis adjustment being an increase or decrease in emphasis; and Wherein the user interface input comprises the interaction with the graphical user interface, and wherein the interaction comprises interaction with a slider corresponding to the one or more emphasized words.

28. A method as claimed in any preceding claim, wherein: The user interface input indicative of the editing of the source NL prompt comprises a spoken input captured in audio data, and the method further comprises: processing the audio data using an automatic speech recognition model to generate recognized text corresponding to the spoken input; and The recognized text is processed to determine the edits to the source NL prompt.

29. A method implemented by one or more processors, the method comprising: Recognize real images captured by real cameras; Identifying a natural language (NL) description of the real image; generating a noise vector for the real image based on the real image using an inversion process; Processing the NL caption using a large-scale language-image (LLI) model and the noise vector to generate a source image that approximates the real image; identifying a source criss-cross attention map produced by using a criss-cross attention layer of the LLI model when generating the source image; identifying a source random seed utilized in generating the source image; After generating the source image: receiving a user interface input indicating an edit to the NL transcript used in generating the source image; In response to receiving the user interface input indicating the edit to the NL transcript: generating an edited image in a plurality of iterations of processing using the LLI model, the edited image being visually similar to the source image but including visual modifications consistent with the edits to the NL transcript indicated by the user interface input, wherein generating the edited image in the plurality of iterations of processing using the LLI model comprises: In the iteration of processing using the LLI model, the process: one or more features generated based on said editing of said source NL transcript, and the source random seed, and At least a portion of the source cross-attention map is injected in at least some of the iterations of generating the edited image using the LLI model.

30. The method of claim 29, wherein: The NL caption for the real image is generated based on other user interface inputs.

31. The method of claim 29, wherein: The NL caption for the real image is automatically generated based on processing the real image using an additional model trained to predict a caption for an image.

32. The method of any one of claims 29 to 31, wherein: The inversion process involves the use of a deterministic denoising diffusion implicit model DDIM.

33. A system comprising a memory storing instructions and one or more processors operable to execute the instructions to perform a method as claimed in any preceding claim.

34. One or more transitory or non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause the method of any one of claims 1 to 32 to be performed.

Images