Image editing from prompt to prompt using cross-attention control

Abstract

Some embodiments are directed to editing a source image, which is generated based on processing a source natural language (NL) prompt using a large language image (LLI) model. Those embodiments edit the source image based on user interface input indicating an edit to the source NL prompt and, optionally, independently of any user interface input specifying a mask for the source image and / or independently of any other user interface input. Some embodiments of the present disclosure additionally or alternatively are directed to applying a prompt-to-prompt editing technique to a source image generated based on and approximating a real image to edit the source image.

Description

[Background technology]

[0001] Large-scale language-image (LLI) models, such as Google's IMAGEN, have demonstrated astonishing semantic and constructive generation capabilities, attracting unprecedented attention from the research community and the public. These LLI models are trained on extremely large language-image datasets and utilize state-of-the-art image generation models, including autoregressive and / or diffusion models. These LLI models enable the generation of images conditioned on plain text, known as text-to-image synthesis. For example, these LLI models can generate a photorealistic image reflecting a dog riding a bicycle in response to the plain text prompt, "a picture of a dog riding a bicycle." Recently, a variety of LLI models have emerged demonstrating unprecedented semantic generation capabilities.

[0002] Image editing is one of the most fundamental tasks in computer graphics, and involves the process of modifying an input image by using auxiliary inputs such as labels, scribbles, masks, or reference images.

[0003] However, many LLI models do not offer simple editing tools for the generated images and generally lack control over the specific semantic domain of a given image (e.g., they only use text guidance). For example, even a slight change in the text prompt can lead to the generation of a completely different output image when using an LLI model. For instance, changing "a picture of a dog riding a bicycle" to "a picture of a white dog riding a bicycle" may result in a completely different generated image, such as an image where the dog's appearance has changed.

[0004] To circumvent this, many proposed LLI-based editing methods require the user to explicitly mask a portion of the image to be repaired, altering only the masked area while matching it to the background of the original image. However, the masking procedure is cumbersome (e.g., requiring a large amount of user input to define the mask) and hinders fast, intuitive text-driven editing. Furthermore, masking image content removes important structural information that is completely ignored during the repair process. Consequently, some editing functions, such as modifying the texture of specific objects, fall outside the scope of repair.

[0005] The specific, intuitive way to edit images is through user-provided text prompts. However, previously proposed LLI-based editing methods may lack the ability to edit images generated via text prompts, or may not have the ability to edit images generated solely via text prompts. [Overview of the project]

[0006] Some embodiments of this disclosure concern editing a source image, which is generated based on processing a source natural language (NL) prompt using a Large Language Image (LLI) model. These embodiments edit the source image based on a user interface input indicating an edit to the source NL prompt, and optionally independently of any user interface input specifying a mask for the source image, and / or independently of any other user interface input. More specifically, these embodiments produce an edited image that is visually similar to the source image but includes visual modifications that match the edit to the source NL prompt. In doing so, the various embodiments can utilize the same random seed(s) used to generate the source image, and furthermore, they can leverage an internal cross-attention map generated when processing the source NL prompt using the LLI model to generate the source image. The cross-attention map is a high-dimensional tensor that links pixels to tokens extracted from the prompt text. For example, various embodiments can control which pixels attend to which tokens of the edited prompt text during which diffusion step by injecting at least some cross-attention maps during at least some iterations of the diffusion process based on the edited prompt.

[0007] Accordingly, various embodiments provide an intuitive image editing interface by editing only the text prompts used when generating the source image (also referred to herein as prompt-to-prompt editing). This enables voice-based, type-based (e.g., physical or virtual keyboard), and / or touch-based (e.g., interaction with emphasis elements, selection of alternative terms) input for editing the source image, eliminating the need for arbitrary specification of image masks and / or other inputs. Such input to editing is natural, can be performed with low latency, and enables a variety of editing tasks that would be difficult in other ways. Furthermore, the embodiments disclosed herein do not require additional, computationally expensive model training, fine-tuning, additional data, or optimization.

[0008] As a non-restrictive example, suppose the source NL prompt is “a furry bear watching a bird,” the source image would reflect a furry bear watching a red bird, and the source image would be generated using an LLI model based on processing “a furry bear watching a bird” and a random seed. Edits to the source NL prompt may include replacing a subset of tokens in the source NL prompt with replacement tokens(s) (e.g., replacing “bird” with “butterfly”), adding tokens(s) to the source NL prompt (e.g., adding “blue” before “bird”), and / or adjusting the emphasis of tokens(s) in the source NL prompt (e.g., increasing emphasis on “fuzzy”).

[0009] Embodiments can generate an edited image by processing at least a portion of a cross-attention map generated when generating a source image in at least some of the features (s) generated based on editing of a source NL prompt, a source random seed, and iterations of processing, using an LLI model. When generating the edited image, by utilizing the cross-attention map in combination with the source random seed, an edited image that is visually similar to the source image but includes visual modifications consistent with the editing is obtained. For example, when "bird" is replaced with "butterfly", the edited image can replace the "red bird" in the source image with "butterfly", but can be very visually similar in other respects. Also, for example, when "blue" is added before "bird", the edited image can replace "red bird" with "blue bird", but can be very visually similar in other respects. As yet another example, when the emphasis in "covered with fur" is increased, the edited image can replace "bear" with "bear covered with more fur" (e.g., more and / or longer fur), but can be very visually similar in other respects. In particular, using the source random seed without utilizing the cross-attention map can result in an image that is not visually similar to the source image.

[0010] Some embodiments of the present disclosure are directed to applying the prompt-to-prompt editing techniques disclosed herein to source images generated based on real images and editing source images that approximate real images. In those embodiments, the first prompt to be edited can be, for example, one specified by an input to a user interface and / or one automatically generated (e.g., using an automatic caption model). Further, in some of those embodiments, the source image is generated by generating a noise vector of the real image (e.g., using an inversion process) and processing an initial prompt using an LLI model and the noise vector to generate a source image that approximates the real image.

[0011] It will be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are intended to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter that appear at the end of this disclosure are intended to be part of the subject matter disclosed herein. Brief Description of the Drawings

[0012] [Figure 1A] Schematically depict exemplary components and interactions that may be involved in generating a source image based on processing a natural language prompt using an LLI model, and in using the LLI model to generate an edited image that is visually similar to the source image but includes visual modifications that match edits to the NL prompt used to generate the source image. [Figure 1B] Schematically depict exemplary components and interactions that may be involved in using an LLI model to generate a source image that approximates a real image, and in using the LLI model to generate an edited image that is visually similar to the source image but includes visual modifications that match edits to the NL prompt used to generate the source image. [Figure 2] Show an exemplary method of generating a source image based on processing a natural language prompt using an LLI model, and storing the random seed(s) used in that processing and the cross-attention maps generated in that processing. [Figure 3] Show an exemplary method of using an LLI model to generate an edited image that is visually similar to the source image but includes visual modifications that match edits to the NL prompt used to generate the source image. [Figure 4]This paper illustrates an exemplary method for generating a source image that approximates a real image by using an LLI model to process natural language prompts and noise vectors for a real image, and by storing the random seed(s) used in the processing and the resulting cross-attention map. [Figure 5] An exemplary computer architecture capable of implementing selected aspects of this disclosure is schematically shown. [Modes for carrying out the invention]

[0013] Before referring to the figures, we provide a non-limiting overview of various embodiments.

[0014] As an example of the various embodiments disclosed herein, I is a source image generated by an LLI model (e.g., a text-guided diffusion model) using prompt P and random seeds s. In some embodiments, the edited prompt P * An attempt is made to edit source image I guided only by [this method], resulting in the edited image I. *This brings about. For example, consider a source image I generated from prompt P "My new bicycle," and assume that the user wants to edit the color and material of the bicycle, or even replace it with a scooter, while preserving the appearance and structure of the source image I. An intuitive interface for the user would be to directly modify the text prompt P by further describing the appearance of the bicycle (e.g., adding "green" before "bicycle"), or by replacing it with another word (e.g., replacing "bicycle" with "scooter"). In contrast to some prior art, the various embodiments disclosed herein avoid relying on any user-defined mask (e.g., a mask defined via interaction with the source image I) to help or represent where editing of the source image I should take place. For example, the various embodiments define "bicycle" in the source image I and avoid relying on any user-defined mask generated based on user interaction with the source image I. Furthermore, in various embodiments disclosed herein, the same random seeds s and edited text prompt P used when generating the source image (I) with the LLI model are used. * It is recognized that processing only (instead of the original text prompt) results in a completely different image with a different structure and composition. For example, (b) edited text prompt P * If the text prompt P is "My new green bicycle" ("green" is added before "bicycle"), then (a) the same random seed s and (b) the edited text prompt P * Processing alone may result in a generated image containing a "green bicycle." However, compared to source image I, such a generated image will have a different structure and composition (e.g., it may contain different background objects).

[0015] In the embodiments disclosed herein, it is recognized that the structure and appearance of the generated image depend not only on the random seed s but also on the pixel - to - pixel interactions for text embedding through the diffusion process. More specifically, in the embodiments disclosed herein, the image editing function from prompt to prompt is enabled by modifying the pixel - to - text interactions that occur in the cross - attention layer, and it is recognized that the structure and composition of the source image I during editing are maintained. More specifically, when generating the image I edited using the LLI model, the composition and structure of the source image I can be retained by injecting at least some of the cross - attention maps generated when generating the source image I. * When generating, by injecting at least some of the cross - attention maps generated when generating the source image I, the composition and structure of the source image I can be retained.

[0016] For additional context regarding cross - attention maps, a specific example of cross - attention in the IMAGEN LLI model (including text - conditioned diffusion models) is described in more detail. Embodiments of IMAGEN include three text - conditioned diffusion models: a 64×64 model from text to image, and two super - resolution models - a 64×64→256×256 model and a 256×256→1024×1024 model. These predict the noise ε θ (z t , c, t) via a U - shaped network for t in the range from T to 1, where z t∫ is a latent vector, and c is a text embedding. More specifically, the 64x64 model starts with a random noise seed and uses a U-Net. The model is conditioned with text embeddings via both cross-attention layers at downsampling and upsampling resolutions within the U-Net, and hybrid attention layers at downsampling and upsampling resolutions within the U-Net. The 64x64 → 256x256 model is conditioned on naively upsampled 64x64 images. An efficient version of the U-Net is used, including a hybrid attention layer at a bottleneck (resolution 32). The 256x256 → 1024x1024 model is conditioned on naively upsampled 256x256 images. An efficient version of the U-Net is used, including only a cross-attention layer at a bottleneck (resolution 64).

[0017] In such IMAGEN LLI models, and / or other LLI models, the composition and geometry are largely determined by the resolution of the text-to-image model's output / initial super-resolution model's input (e.g., 64x64 in the example above). Therefore, some embodiments can use the super-resolution process directly when generating the edited image, performing the fitting only in the text-to-image diffusion process. When generating an image using an IMAGEN LLI model, and / or other LLI models, each diffusion step or iteration t uses a U-shaped network to process the noisy image z t This includes predicting noise f from the text embedding ψ(P). In the final diffusion step, the image I=z0 generated by this process is obtained. Notably, the interaction between the two modalities occurs during noise prediction, where the embeddings of visual and textual features are fused using a cross-attention layer that produces a spatial attention map for each text token.

[0018] More formally, a noisy image Φ(z tThe deep space features of ) are the query matrix Q=l Q (Φ(z t The text embedding is projected onto the learned linear projection l Q ,l K ,l V Through this, the key matrix K=l K (ψ(P)) and the value matrix V=l V It is projected onto (ψ(P)). Next, the attention map is,

number

number

[0019] Intuitively, the cross-attention output MV is a weighted average of values ​​V, where the weights are attention maps M correlated to the similarity between the query matrix Q and the key matrix K. In practice, to increase their expressiveness, multi-head attention can be used in parallel, and then the results are concatenated and passed through a trained linear layer to obtain the final output.

[0020] IMAGEN and / or other LLI models(s) conditioned on text prompts in noise prediction at each diffusion step through two types of attention layers: i) a cross-attention layer and ii) a hybrid attention layer that acts as both self-attention and cross-attention by concatenating text embedding sequences to key-value pairs in each self-attention layer. Both can be called cross-attention because various embodiments can intervene only in the cross-attention portion of the hybrid attention; that is, only the last channel referencing the text token is modified by the hybrid attention module.

[0021] When controlling cross-attention in the IMAGEN LLI model and / or other LLI models, please note again that the spatial layout and geometry of the generated image depend on the cross-attention map generated during the image's creation. This interaction between pixels and text can be observed from a plot of the average attention map generated during image creation. In such a plot, it can be observed that pixels are attracted to the words that describe them. For example, in a prompt containing the word "bear," pixels depicting bears can be observed to correlate with the word "bear." Such observations indicate that the image structure is already determined in the early steps of the diffusion process.

[0022] Since attention reflects the overall composition, the attention map M obtained from generating the source image I using the original prompt P and LLI model is used in the edited image I. * When generating the corrected prompt P * This can then be injected into a second generation using the LLI model. This results in an edited image I that not only operates according to the edited prompts but also retains the structure of the input image I. *This enables the synthesis of different types of edits. Such examples are concrete examples of a broader set of attention-based operations that enable different types of intuitive editing. Therefore, the following paragraphs will describe a more general framework, followed by details of various specific editing operations.

[0023] DM(z t ,P,t,s) is a noisy image z t-1 and attention map M t This is the calculation of a single step t of the diffusion process that outputs (omitted if not used).

number

number

number

number

[0024] A common algorithm for controlled image generation may involve performing an iterative diffusion process simultaneously for both prompts, where attention-based operations are applied to each step according to the desired editing task. The internal randomness used in each diffusion process, which may be reflected by the random seed(s), can be fixed / same in each process. This is due to the nature of the diffusion model, which causes two random seeds to produce significantly different outputs, even with the same prompt. Formally, our common algorithm is as follows:

[0025] More formally, a general algorithm for various embodiments may be as follows:

number

[0026] Note that in the algorithm described above, the image I generated by prompt P and random seed s may be defined as an additional input. Nevertheless, the algorithm would remain the same. Also note that in the algorithm described above, the forward call on line 7 may be skipped by applying the editing function within the diffuse forward function. Additionally or alternatively, the diffuse step may be z t-1 and

number

[0027]

number

[0028] In word swapping, a user interface input is provided that indicates the user has swapped one token(s) from the original prompt with another. For example, if the user interface input indicates an edit from the original prompt "big red bicycle" to the edited prompt "big red car," then "bicycle" may be swapped with "car." Such user interface input may be via touch and / or type input deleting "bicycle" and typing "car," and / or via speech user interface input (e.g., speech input "replace bicycle with car"). In word swapping and / or other editing operations, the challenge is to address the content of the edited prompt while preserving the original composition. To this end, embodiments inject the attention map generated when generating the source image into the generation of the edited image using the edited prompt. However, the proposed attention injection may over-constrain the geometry, especially when large structural modifications such as "bicycle" to "car" are involved. Such over-constraining of geometry can be addressed in some embodiments of word swapping editing by softer attention constraints. For example, a softer attention constraint can be expressed using the following editing function:

number

[0029] In the aforementioned editing function, τ is the timestamp / iteration parameter, which determines up to which step the injection is applied. Note that composition is determined in the early steps of the diffusion process. Therefore, by limiting the number of injection steps, the composition of the newly generated image can be guided while still allowing the necessary geometric degrees of freedom to fit the new prompt. Additional or alternative fitting is to assign different numbers of injection timestamps to different tokens in the prompt. If two words are represented using different numbers of tokens, the map can be duplicated / averaged as needed using an alignment function, such as the one described for adding new phrases.

[0030] When adding a new phrase, a user interface input is provided that indicates the user has added a new token(s) to the original prompt. For example, “a drawing by children” can be added before the original prompt “a castle next to a river,” in which case the user interface input indicates such a preceding addition. For example, the user interface input could include typing “a drawing by children” at the beginning of the original prompt, or it could be a spoken user interface input such as “adding a drawing by children, before it.” When adding a new phrase, in order to preserve common details, embodiments may apply attention injection only to common token(s) from both prompts. For example, attention injection may be applied only to “a castle next to a river” in the above example. More formally, an alignment function A may be used, which is applied to the edited prompt P * Receive the token index from P * In this case, output the corresponding token index, or output nothing if there is no match. For such an alignment function, an exemplary editing function could be represented as follows:

number

[0031] In the aforementioned editing function, it should be noted that index i corresponds to a pixel value and j corresponds to a text token. Optionally, the aforementioned editing function, as with the word-exchange editing function, allows setting a timestamp τ that controls the number of diffusion steps to which the injection is applied. Such editing functions enable a variety of prompt-to-prompt functionality, such as stylization, specifying object attributes, or global operations.

[0032] Attention reweighting provides user interface inputs that indicate the user's desire to increase or decrease the degree to which the token(s) of the original prompt affect the original source image. For example, the original prompt might be "fluffy red ball," and the user might want an edited image where the ball is fluffier or less fluffy than in the original image. User interface inputs indicating such an increase or decrease in fluffiness could be, for example, interaction with a slider or up and down arrows presented alongside "fluffy," "fluffy" in bold or underlined, and / or verbal input (e.g., "more fluffy"). Attention reweighting of the token(s) of the original prompt uses a scaling parameter c to assign token(s) j corresponding to the token(s) to which the emphasis user interface input is directed. * An attention map. For example, the scaling parameter c can be a negative parameter if the emphasis input indicates a decrease, and a positive parameter if the emphasis input indicates an increase, and optionally has a magnitude based on the degree of increase or decrease indicated by the emphasis input. For example, the scaling parameter c can be represented as c∈[-2,2]. The rest of the attention map can remain unchanged. Such an editing function is,

number

[0033] Next, we provide several non-limiting examples of practical applications of various embodiments, demonstrating that intuitive text-only editing is possible by controlling the spatial layout corresponding to each word in the user-provided prompts.

[0034] One practical application is the local editing of a source image, which does not require any user-provided masks, by editing a user-provided source prompt. For example, a source image can be generated using the prompt "lemon cake" and an LLI model. The user interface input can replace "lemon" with "pumpkin," resulting in an edited prompt of "pumpkin cake." By utilizing the embodiments disclosed herein, an edited image can be generated that preserves the spatial layout, geometry, and semantics of the source image. On the other hand, naively providing the prompt "pumpkin cake" to a synthesis model will result in entirely different geometry, even when using the same random seed in a deterministic setting.

[0035] Other practical applications include performing structural modifications on a source image in addition to, or instead of, modifying only the texture. For example, a source image can be generated using a prompt and LLI model that includes "bicycle" (among other words), and a user interface input can replace "bicycle" with "car". By utilizing the embodiments disclosed herein, an edited image can be generated in which "bicycle" in the source image is changed to "car" in the edited image. It has been observed that the more diffusion steps to which cross-attention injection is applied when generating the edited image, the higher the fidelity to the original image. However, the optimal result is not necessarily achieved by applying injection to all diffusion steps. Therefore, cross-attention injection can be optionally applied to only a subset of steps or iterations, such as a threshold percentage between 5% and 95%, 15% and 90%, or other boundaries. Optionally, an interactive user interface element may be presented along with the edited prompt, allowing user input to define the degree of fidelity to the original image that should be adhered to when generating the edited image. If such user interface elements are provided, a subset of the steps or iterations to which cross-attention injection is applied can correspond to user interface inputs directed to those interactive user interface elements (if any). For example, the interactive user interface elements may include sliders, and the number of iterations to which cross-attention injection is applied can be based on the position of the sliders.

[0036] Another practical use is when, instead of substituting one word for another, the user may want to add a new specification to the generated source image. For example, the generated source image may be generated based on the source prompt "car on the side of the street," and a user interface input may be provided to add "crushed" before the car, resulting in the edited prompt "crushed car on the side of the street." In such a case, the attention map of the source prompt can be used when generating the edited image, and at the same time, the newly added word ("crushed") and its corresponding attention map can also be used when generating the edited image. This can result in an edited image that includes a crushed car (which the source image did not) while still retaining the background of the source image.

[0037] Another practical use is to perform global editing while preserving the image composition of the source image. In such applications, the editing affects all parts of the image, but the original composition, such as the position and identity of objects, should still be preserved. For example, by editing the source prompt "car on the side of the street" to "car on a snow-covered street," snow can be added to the background and car while preserving the background and car of the source image. As another example, by editing the source prompt "photograph of a waterfall" to "impression of a waterfall," the original composition of the source image can be preserved while changing it from a photograph to an impression.

[0038] While various embodiments of this disclosure describe applying prompt-to-prompt editing techniques to a source image that is generated by processing source prompts using an LLI model, embodiments of this disclosure additionally or alternatively concern applying the prompt-to-prompt editing techniques disclosed herein to editing a source image that approximates a real image, where the source image is generated based on a real image (e.g., a real image captured by a real-world physical camera). In those embodiments, the initial prompt to be edited may be, for example, specified by user interface input and / or automatically generated (e.g., using an autocapture model). Furthermore, in some of those embodiments, the source image is generated by generating noise vectors of a real image (e.g., using an inversion process) and processing initial prompts using the LLI model and noise vectors to generate a source image that approximates a real image.

[0039] In embodiments applying prompt-to-prompt editing techniques to editing source images, it is recognized that editing a real image may require finding an initial noise vector that, when given to a diffusion process, produces a given input image. This process is generally known as inversion, but has not been conventionally utilized in LLIs such as text-guide diffusion models. A naive approach would be to add Gaussian noise to the real image and then perform a predefined number of diffusion steps. However, such an approach can result in significant distortion. Therefore, some embodiments disclosed herein employ an improved inversion approach based on a deterministic denoising diffusion implicit model (DDIM) rather than a denoising diffusion stochastic model (DDPM). These embodiments are in the reverse direction, i.e., x T →x0 instead of x0→x T The diffusion process can then be performed, where x0 is set to be a real image.

[0040] Such inversion processes can yield satisfactory results. However, in many other cases, such inversions are not sufficiently accurate. This may be partly due to a distortion-editability trade-off, where reducing the guidance parameters without a classifier (i.e., reducing the influence of prompts) improves the reconstruction but limits its ability to perform significant operations. To mitigate this limitation, some embodiments use masks extracted directly from the attention map to restore the unedited areas of the original image. It should be noted that in those embodiments, the masks are generated without user guidance. Furthermore, in some of these embodiments, this approach can also work using a naive DDPM inversion scheme (adding noise and then denoising).

[0041] DDPM approximates the data distribution q(x0) and the distribution p from which sampling is easy. θ DDPM is a generative latent variable model aimed at modeling (x0). DDPM models the "forward process" in the space of x0 from data to noise.

[0042] This process is a Markov chain starting from x0, with latent variables x1, ..., x T Noise is gradually added to the data to generate ∈X. Therefore, the sequence of latent variables is:

number

number

[0043] An interesting property of the forward process is that, without sampling the intermediate latent vector, the latent variable x is directly generated as the following linear combination of noise and x0. t The ability to express:

number

number

[0044] To sample from the distribution q(x0), an isotropic Gaussian noise x T A double "reverse process" from data p(x t-1 |x t ) is posterior q(x t-1 |x t Defined by sampling ). A computationally intractable reverse process q(x t-1 |x t ) depends on an unknown data distribution q(x0), therefore it is a parameterized Gaussian transition network.

number

[17]

[0045] Under this definition,

number

[0046] trained ε θ (x t If t) is present, the following sample method can be used: x t-1 =μ θ (xt , t) + σ t z, z ~ N(0, I). The σ of each sample stage t can be controlled. In DDIM, σ t = 0 can be used to deterministically determine the sampling process. The reverse process is finally trained by solving the following optimization problem: [Number] The parameter θ can be taught to fit q(x0) by maximizing the variational lower bound.

[0047] Here, referring to the drawings, FIG. 1A shows generating a source image 103A based on processing a natural language prompt 101A using an LLI model 150, and using the LLI model 150 to generate an edited image 107A that is visually similar to the source image 103A but includes visual corrections that match the edits to the NL prompt used to generate the source image 103A as reflected in the prompt editing input 106A. Schematically depicts exemplary components and interactions that may be involved.

[0048] In FIG. 1A, the client device 110 can provide an NL prompt 101A, such as an example of "a photo of an orange cat riding a bicycle". The NL prompt 101A can be generated based on user interface inputs provided by the user on the client device 110, such as typed inputs or spoken inputs. For example, the NL prompt 101A can be based on text from speech recognition executed based on a spoken input received on the client device 110.

[0049] The source image engine 120 can process the NL prompt 101A using the LLI model 150 to generate the source image 103A. When generating the source image 103A, one or more random (true random or pseudo-random) seeds 104A can be used. In addition, a cross-attention map 105A is generated when generating the source image 103A. The random seed(s) 104A and the cross-attention map 105A can be provided to the editing image engine 130.

[0050] The editing image engine 130 receives a prompt editing input 106A, which is a user interface input provided by the client device 110, specifying one or more edits to the NL prompt 101A, such as a replacement input (e.g., replace "bicycle" with "horse"), an append input (e.g., add "green" before "bicycle"), and / or an emphasis adjustment input (e.g., increase the emphasis of "orange"). In response to receiving the prompt editing input 106A, the editing image engine 130 can interact with the LLI model 150 when generating an edited image 107A that is visually similar to the source image 103A but includes visual modifications that match the edit(s) to the NL prompt 101A, as reflected by the prompt editing input 106A.

[0051] When interacting with the LLI model 150 to generate the edited image 107A, the edited image engine 130 may utilize the random seed(s) 104A used to generate the source image 103A, utilize edit features based on edits reflected by prompt edit inputs 106A (e.g., text embedding of modified prompts reflected by the edits), and utilize at least some of the cross-attention maps 105A in at least some of the iterations generating the edited image 107A. Which cross-attention maps 105A are used to generate the edited image 107A, and / or which iterations the cross-attention maps are used in, may depend on the type(s) of edits reflected by prompt edit inputs 106A (e.g., whether the edits are of the replacement, addition, or enhancement type).

[0052] Figure 1B schematically depicts exemplary components and interactions that may be involved in using the LLI model 150 to generate a source image 103B that approximates the real image 102B, and in using the LLI model 150 to generate an edited image that is visually similar to the source image 103B but includes visual modifications that match the edits made to the NL prompt used to generate the source image 103B, which are reflected in the prompt editing input 106B.

[0053] In Figure 1B, the client device 110 can provide the real image 102B to the noise vector engine 120. The noise vector engine 120 can generate a noise vector 102B1 for the real image 102B. For example, the noise vector engine 120 can generate the noise vector 102B1 using an inversion process and the real image, such as by using a DDIM or DDPM inversion process. The noise vector 102B1 is provided to the source image engine 120 along with an NL prompt 101B for the real image 102B. The NL prompt 101B can be provided by the client device, based on user interface input (e.g., a user-curated caption for the real image 102B), and / or provided by a caption engine 140 that automatically generates the NL prompt by processing the real image 102B using a caption model.

[0054] The source image engine 120 can process the NL prompt 101B and noise vector 102B1 using the LLI model 150 to generate a source image 103B that approximates the real image 102B. When generating the source image 103B, one or more random (true random or pseudo-random) seeds 104B can be used. Furthermore, a cross-attention map 105B is generated when generating the source image 103B. The random seed(s) 104B and the cross-attention map 105B can be provided to the editing image engine 130.

[0055] The editing image engine 130 receives a prompt editing input 106B, which is a user interface input provided by the client device 110, specifying one or more edits to the NL prompt 101B (which can optionally be rendered on the client device 110 based on the output from the caption engine 140), such as a replacement input, an append input, and / or an enhancement adjustment input. In response to receiving the prompt editing input 106B, the editing image engine 130 can interact with the LLI model 150 when generating an edited image 107B that is visually similar to the source image 103B but includes visual modifications that match the edit(s) to the NL prompt 101B, as reflected by the prompt editing input 106B.

[0056] When interacting with the LLI model 150 to generate the edited image 107B, the edited image engine 130 may utilize the random seed(s) 104B used to generate the source image 103B, utilize edit features based on edits reflected by prompt edit inputs 106B (e.g., text embedding of modified prompts reflected by the edits), and utilize at least some of the cross-attention maps 105B in at least some of the iterations generating the edited image 107B. Which cross-attention maps 105B are used to generate the edited image 107B, and / or which iterations the cross-attention maps are used in, may depend on the type(s) of edits reflected by prompt edit inputs 106B (e.g., whether the edits are of the replacement, append, or enhancement type).

[0057] Figure 2 illustrates an exemplary method 200 for generating a source image based on processing a natural language prompt using an LLI model, and for storing the random seed(s) used in the processing and the resulting cross-attention map. For convenience, the operation of the flowchart is described with reference to the system performing the operation. This system may include various components of various computer systems, such as one or more components of a server computing device(s). Furthermore, the operation of method 200 is shown in a specific order, but this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0058] In block 202, the system receives a natural language prompt. For example, the natural language prompt may be provided based on user interface input on a client device, such as user interface input directed to the system interface or front end, which is accessible via the client device.

[0059] In block 204, the system generates one or more source random seeds. For example, the system can use a random process or a pseudo-random process to generate the source random seed(s).

[0060] In block 206, the system generates a source image based on processing the source random seed in block 204 and the NL prompt in block 202 using the LLI model. A cross-attention map is generated when generating the source image based on processing using the LLI model, as described herein. The cross-attention map may contain values ​​that associate tokens from the NL prompt with pixels in the generated source image.

[0061] In block 208, the system stores (for example, at least temporarily in memory) the random seed(s) from block 204 and the cross-attention map generated during the source image generation in block 206.

[0062] In block 210, the system triggers the rendering of the source image and the NL prompt 210. For example, the system can trigger such rendering on a client device that provided the natural language prompt in block 202.

[0063] Figure 3 illustrates an exemplary method 300 that uses an LLI model to generate an edited image that is visually similar to the source image but includes visual modifications that match the edits made to the NL prompts used to generate the source image. For convenience, the operations of the flowchart are described with reference to the system that performs the operations. This system may include various components of various computer systems, such as one or more components of a server computing device. Furthermore, although the operations of method 300 are shown in a specific order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0064] In block 302, the system receives a user interface input indicating an edit to the source NL prompt used when generating the source image. The user interface input may be received on the client device in response to the rendering of the source image and, optionally, in response to the rendering of the NL prompt used when generating the source image. The source image may be the source image from block 206 of the iteration of method 200 in Figure 2, and the NL prompt may be the NL prompt from block 206 of the iteration of method 200 in Figure 2. Alternatively, the source image may be the (approximately real image) source image from block 410 of the iteration of method 400 in Figure 4, and the NL prompt may be the NL prompt from block 404 of the iteration of method 400 in Figure 4.

[0065] In various embodiments, block 302 includes one or more of the subblocks 302A, 302B, and 302C. In subblock 302A, the user interface for block 302 input includes a substitution input. The substitution input can reflect an edit which is the substitution of a subset of tokens in the source NL prompt with one or more substitution tokens that are different from the subset of tokens in the source NL prompt. In subblock 302B, the user interface for block 302 input includes an append input. The append input can reflect an edit which is the appending of one or more append tokens to the source NL prompt. In subblock 302A, the user interface for block 302 input includes an emphasis adjustment input. The emphasis adjustment input can reflect an edit which is an adjustment of emphasis in one or more emphasis tokens in the source NL prompt, where the adjustment is an increase or decrease in emphasis and can optionally reflect the magnitude of the increase or decrease.

[0066] In block 304, the system generates edit features based on edits to the source NL prompt, which are reflected by the user interface input received in block 302. For example, if the edit is a substitution, the system may generate an edit feature that includes text embedding of the modified prompt conforming to the source NL prompt, but replacing a subset of the tokens in the source NL prompt with the edited tokens. As another example, if the edit is an addition, the system may generate an edit feature that includes text embedding of the modified prompt, including the source NL prompt and the additional tokens. As yet another example, if the edit is an adjustment of emphasis on one or more emphasis tokens, the system may generate an edit feature that includes scaled attention maps for one or more emphasis tokens.

[0067] In block 306, the system identifies the source seed(s) and cross-attention map used to generate the source image in block 302. The source seed(s) and cross-attention map may be those used to generate the source image in an iteration of method 200 in Figure 2, or those used to generate a source image (approximating a real image) in an iteration of method 400 in Figure 4.

[0068] In block 308, the system generates an edited image based on processing (A) edit features generated based on edits to the source NL prompt (generated in block 304), (B) source seed(s) (identified in block 306), and (C) at least some of the cross-attention maps (identified in block 306) using the LLI model.

[0069] In some embodiments, block 308 includes a subblock 308A in which the system uses only a subset of the cross-attention map and / or uses the cross-attention map only in a subset of iterations of processing. In some versions of those embodiments, whether a subset of the cross-attention map is used, or which subset of the cross-attention map is used, may depend on edits to the source NL prompt reflected by user interface input received in block 302. Furthermore, in some versions of those, or other versions of those embodiments, whether the cross-attention map applies only to a subset of iterations and / or which subset the cross-attention map applies to may depend on edits to the source NL prompt reflected by user interface input received in block 302. For example, if the edit is a substitution, only a subset of the cross-attention map that excludes those corresponding to the substituted token(s) may be used, and this subset may be used only in a subset of iterations. As another example, if the edit is an addition, the cross-attention map can optionally contain all of the cross-attention maps, but they can only be used in a subset of the iterations (for example, they cannot be used in processing features corresponding to replacement tokens). As yet another example, if the edit is an adjustment of emphasis in emphasis tokens, the first subset of the cross-attention map can be used for non-emphasized tokens, and a scaled version of the second subset of the cross-attention map can be used for emphasis tokens.

[0070] In some additional or alternative embodiments, subblock 308A may include a system that always uses a cross-attention map only for a subset of iterations of processing, such as only for a threshold percentage of iterations. For example, the threshold could be between 5% and 95%, between 15% and 90%, between 25% and 75%, or between any other boundary(s). Optionally, in some versions of those additional or alternative embodiments, an interactive user interface element(s) may be presented, allowing user input to define the degree of fidelity to the original image that should be adhered to when generating the edited image. In some versions of those, the threshold may be determined by the system based on interaction(s) with the user interface element(s).

[0071] In block 310, the system triggers the rendering of the edited image and, optionally, the edited NL prompt. For example, the system can trigger such rendering on a client device that provided the user interface input in block 302.

[0072] In the optional block 312, the system may indicate further edits to the source NL prompt and monitor for new user interface inputs to be added to the edit(s) of the previous iteration(s) of block 302. If such new user interface inputs are detected, the system may proceed to execute other iterations of blocks 302, 304, 306, 308, and 310 based on such new user interface inputs.

[0073] Figure 4 illustrates an exemplary method 400 for generating a source image that approximates a real image by using an LLI model to process natural language prompts and noise vectors for a real image, and by storing the random seed(s) used in the processing and the resulting cross-attention map. For convenience, the operations of the flowchart are described with reference to the system performing the operations. This system may include various components of various computer systems, such as one or more components of a server computing device(s). Furthermore, the operations of method 400 are shown in a specific order, but this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0074] In block 402, the system identifies real images captured by real cameras, such as real images uploaded from a client device.

[0075] In block 404, the system identifies the NL prompt for the real image. When identifying the NL prompt for the real image, the system may execute subblock 404A or subblock 404B.

[0076] In subblock 404A, the NL prompt for the real image is generated based on user interface input. For example, when a real image is received from a client device in block 402, an NL prompt may also be received and can respond to user interface input received by the client device. For example, user interface input may be received by the client device in response to the rendering of a prompt such as "Please provide a natural language description of this image."

[0077] In subblock 404B, NL prompts for real images are generated based on processing the real images using a caption model or other visual language model.

[0078] In block 406, the system generates noise vectors for the real image. For example, the system can generate noise vectors based on applying an inversion process, such as a DDIM or DDPM inversion process, to the real image.

[0079] In block 408, the system generates one or more source random seeds. For example, the system may use a random process or a pseudo-random process to generate the source random seed(s).

[0080] In block 410, the system uses an LLI model to generate a source image that approximates a real image by processing the source random seed(s) in block 408, the noise vector in block 406, and the NL prompt in block 404. When generating the source image based on such processing using the LLI model, a cross-attention map arises, as described herein. The cross-attention map may contain values ​​that link tokens from the NL prompt to pixels in the generated source image.

[0081] In block 412, the system stores (for example, at least temporarily in memory) the random seed(s) from block 408 and the cross-attention map generated during the source image generation in block 410.

[0082] In the optional block 414, the system triggers the rendering of the source image and / or the NL prompt 210. For example, the system may trigger such rendering on a client device that provided the real image in block 402.

[0083] Figure 5 is a block diagram of an exemplary computing device 510 that can be optionally used to implement one or more embodiments of the techniques described herein. For example, all or embodiments of computing device 510 can be incorporated into a server(s) or other computing device(s) used to implement the prompt-to-prompt editing techniques disclosed herein.

[0084] The computing device 510 typically includes at least one processor 514 that communicates with numerous peripheral devices via a bus subsystem 512. These peripheral devices include, for example, a storage subsystem 524 including a memory subsystem 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices enable user interaction with the computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to a corresponding interface device in another computing device.

[0085] User interface input devices 522 include pointing devices such as keyboards, mice, trackballs, touchpads, and graphics tablets; audio input devices such as scanners, touchscreens integrated into displays, and speech recognition systems; microphones; and / or other types of input devices. In general, the use of the term “input device” is intended to include all possible types of devices and methods for inputting information into the computing device 510 or a communication network.

[0086] User interface output devices 520 may include non-visual displays such as display subsystems, printers, fax machines, or audio output devices. Display subsystems may include flat panel devices such as cathode ray tubes (CRTs) and liquid crystal displays (LCDs), projection devices, or other mechanisms for creating visible images. Display subsystems may also provide non-visual displays via audio output devices, etc. Generally, the use of the term “output device” is intended to include all possible types of devices and methods for outputting information from computing device 510 to a user or other machine or computing device.

[0087] The storage subsystem 524 stores programming and data structures that provide some or all of the functionality of the modules described herein. For example, the storage subsystem 524 may include logic to perform selected embodiments of the methods in Figures 2, 3, and / or 4, as well as logic to implement the various components described herein.

[0088] These software modules typically run on processor 514 alone or in combination with other processors. The memory 525 used by the storage subsystem 524 may include a number of memories, such as main random access memory (RAM) 530 for storing instructions and data during program execution, and read-only memory (ROM) 532 for storing fixed instructions. The file storage subsystem 526 can provide persistent storage of program files and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of one embodiment may be stored by the file storage subsystem 526 within the storage subsystem 524 or on other machines accessible by the processor 514(or more).

[0089] The bus subsystem 512 provides a mechanism that enables various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is schematically shown as a single bus, multiple buses can be used in alternative embodiments of the bus subsystem.

[0090] The computing device 510 can be of various types, such as a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Because computers and networks are constantly changing, the description of the computing device 510 shown in Figure 5 is intended only as a specific example to illustrate several embodiments. The computing device 510 can have many other configurations, some with more or fewer components than the computing device shown in Figure 5.

[0091] While several embodiments are described and illustrated herein, various other means and / or structures can be utilized to perform the functions described herein and / or to obtain the results and / or one or more advantages, and each such variation and / or modification is considered to fall within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials and configurations described herein are for illustrative purposes only, and the actual parameters, dimensions, materials and / or configurations will depend on the specific application in which the teaching(s) are used. Those skilled in the art will be able to recognize or verify many equivalents to the specific embodiments described herein by performing ordinary experiments. Thus, it should be understood that the embodiments described herein are presented for illustrative purposes only, and embodiments may be practiced in ways other than those specifically described and claimed, within the scope of the appended claims and their equivalents. Embodiments of this disclosure cover the individual features, systems, articles, materials, kits and / or methods described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included in the scope of this disclosure, provided that they do not conflict with each other.

[0092] In some embodiments, a method is provided which is implemented by a processor(s), the method includes identifying a source cross-attention map generated using the cross-attention layer of a Large Language Image (LLI) model when generating a source image based on processing a source natural language (NL) prompt using an LLI model. The method further includes identifying one or more source random seeds used when generating the source image based on processing a source NL prompt using an LLI model. After generating the source image, the method further includes receiving a user interface input indicating edits to the source NL prompt used when generating the source image. In response to receiving a user interface input indicating edits to the source NL prompt, the method further includes generating an edited image over multiple iterations of processing using an LLI model that is visually similar to the source image but includes visual modifications that match the edits to the NL prompt indicated by the user interface input. In an iteration of processing using the LLI model, generating an edited image may include processing one or more features generated based on editing the source NL prompt and the source random seed, and in at least some of the iterations of generating an edited image using the LLI model, injecting at least a portion of the source cross-attention map.

[0093] These and other embodiments of the technology disclosed herein may include one or more of the following features:

[0094] In some embodiments, editing involves replacing a subset of tokens in the source NL prompt with one or more replacement tokens that are different from the subset of tokens in the source NL prompt. In some versions of those embodiments, one or more features generated based on editing the source NL prompt include text embeddings of a modified prompt that conform to the source NL prompt, but replace a subset of tokens in the source NL prompt with edited tokens. In some versions of those embodiments, in at least some of the iterations of generating the edited image using the LLI model, injecting at least a portion of the source cross-attention map includes using the entire source cross-attention map when processing the text embeddings of the modified prompt in at least some of the iterations. In some variations of those versions, at least some of the iterations are a subset of the iterations, and in other iterations not included in the subset of the iterations, other cross-attention maps are used when processing text embeddings, while the source cross-attention map is not used when processing text embeddings. For example, a subset of iterations can be an initial consecutive sequence of iterations, and may contain more than 5 percent of the iterations but less than 95 percent of the iterations, and / or more than 10 percent of the iterations but less than 90 percent of the iterations.

[0095] In some embodiments, editing involves adding one or more additional tokens to a source NL prompt. In some versions of those embodiments, one or more features generated based on editing to a source NL prompt include text embedding of the modified prompt, including the source NL prompt and the additional tokens. In some variations of those versions, in at least some of the iterations of generating the edited image using the LLI model, injecting at least a portion of the source cross-attention map involves using the entire source cross-attention map when processing portions of the text embedding corresponding to the source NL prompt, while the source cross-attention map is not used when processing additional portions of the text embedding corresponding to the additional tokens. In some variations of those embodiments, at least some of the iterations are a subset of the iterations, and in other iterations not included in the subset, the source cross-attention map is not used when processing portions of the text embedding corresponding to the source NL prompt. For example, a subset of iterations could be an initial consecutive sequence of iterations, more than 5 percent of the iterations but less than 95 percent of the iterations, and / or more than 20 percent of the iterations but less than 75 percent of the iterations.

[0096] In some embodiments, editing involves adjusting emphasis in one or more emphasis tokens among the source tokens of a source NL prompt, where emphasis adjustment is an increase or decrease in emphasis. In some versions of those embodiments, one or more features generated based on editing to a source NL prompt include one or more scaled attention maps for one or more emphasis tokens, and the method further includes identifying emphasis portions of a source cross attention map corresponding to one or more emphasis tokens, and generating one or more scaled attention maps by scaling the emphasis portions in response to emphasis adjustments. In some variations of those versions, emphasis adjustment is an increase in emphasis, and generating one or more scaled attention maps by scaling the emphasis portions in response to emphasis adjustments includes increasing the value of the emphasis portions by a coefficient. In some variations of those variations, the increase in emphasis indicated by a user interface input is of a specific magnitude, which is one of several candidate magnitudes, where the coefficient is proportional to the specific magnitude. In some embodiments that optionally include editing, which is the adjustment of emphasis in emphasis tokens(s), the text embedding of the source NL prompt is processed by iterating through processing using an LLI model, and the text embedding includes emphasis embedding portions corresponding to one or more emphasis tokens and rest embeddings corresponding to the rest of the source NL prompt after the emphasis portions have been removed.Furthermore, optionally, at least a portion of the source cross-attention map is the remaining portion of the source cross-attention map after the emphasis portion has been removed, and in at least some of the iterations of generating the edited image using the LLI model, injecting at least a portion of the source cross-attention map includes using the remaining portion of the source cross-attention map when processing the embedding of the remaining portion in at least some of the iterations, where one or more scaled source cross-attention maps are used when processing the emphasis embedding portion in at least some of the iterations.

[0097] In some embodiments, at least some of the iterations are all of the iterations.

[0098] In some embodiments, the cross-attention map includes values ​​that associate tokens of the NL prompt with pixels in the source image. In some of those embodiments, each value defines the corresponding weight of the corresponding tokens in the corresponding pixels of the pixel.

[0099] In some embodiments, the method further includes generating a source image based on processing a source natural language (NL) prompt using an LLI model.

[0100] In some embodiments, a user interface input indicating editing a source NL prompt includes interaction with a graphical user interface that renders the typed input and / or the source NL prompt. In some versions of those embodiments, the editing includes adjusting emphasis in one or more emphasis tokens among the source tokens of the source NL prompt, and the emphasis adjustment is increasing or decreasing emphasis. In some versions of those embodiments, the user interface input includes interaction with a graphical user interface, and the interaction includes interaction with sliders corresponding to one or more emphasis tokens.

[0101] In some embodiments, a user interface input indicating an edit to a source NL prompt includes speech input captured in audio data. In some of those embodiments, the method further includes processing the audio data to generate recognized text corresponding to the speech input using an automatic speech recognition model, and processing the recognized text to determine an edit to the source NL prompt.

[0102] In some embodiments, a method is provided that is implemented by a processor(s), the method comprising: identifying a real image captured by a real camera; and identifying a natural language (NL) caption for the real image. The method further comprises generating a noise vector for the real image using an inversion process and based on the real image. The method further comprises processing the NL caption to generate a source image that approximates the real image using a Large Language Image (LLI) model and the noise vector. The method further comprises identifying a source cross-attention map generated using the cross-attention layer of the LLI model when generating the source image. The method further comprises identifying a source random seed used when generating the source image. After generating the source image, the method further comprises receiving a user interface input indicating edits to the NL caption used when generating the source image. In response to receiving a user interface input indicating edits to the NL caption, the method further comprises generating an edited image in multiple iterations of processing using the LLI model that is visually similar to the source image but includes visual modifications that match the edits to the NL caption indicated by the user interface input. In multiple iterations of processing using the LLI model, generating an edited image may include processing one or more features generated based on editing the source NL caption and the source random seed in each iteration of processing using the LLI model, and injecting at least a portion of the source cross-attention map in at least some of the iterations of generating the edited image using the LLI model.

[0103] These and other embodiments of the technology disclosed herein may include one or more of the following features:

[0104] In some embodiments, NL captions for real images are generated based on other user interface inputs.

[0105] In some embodiments, nonlinear captions for real images are automatically generated based on processing the real images using an additional model trained to predict captions for images.

[0106] In some embodiments, the inversion process involves using a deterministic denoising diffusion implicit model (DDIM).

[0107] Other embodiments may include a non-temporary computer-readable storage medium that stores instructions executable by one or more processors (e.g., a central processing unit (CPU) (or multiple CPU), a graphics processing unit (or multiple GPU) (or multiple GPU)), and / or tensor processing units (or multiple TPU)) for carrying out one or more of the methods described herein. Further embodiments may include a system of one or more computers, each containing one or more processors operable to execute the stored instructions, for carrying out one or more of the methods described herein.

Claims

1. A method implemented by one or more processors, wherein the method is When generating a source image based on processing a source natural language (NL) prompt using a large-scale language image (LLI) model, the source cross-attention map generated using the cross-attention layer of the LLI model is identified, Based on processing the source NL prompt using the LLI model, one or more source random seeds used when generating the source image are identified. Following the generation of the aforementioned source image, Receiving user interface input indicating edits to the source NL prompt used when generating the source image, In response to receiving the user interface input indicating the edit to the source NL prompt, The process includes, in multiple iterations of the process using the LLI model, generating an edited image that is visually similar to the source image but includes visual modifications that match the edits to the source NL prompt indicated by the user interface input, and in the iterations of the process using the LLI model, generating the edited image is In the iteration of the process using the LLI model, One or more features generated based on the editing of the source NL prompt, The aforementioned source random seed and the process of, In at least some of the iterations of generating the edited image using the LLI model, at least a portion of the source cross-attention map is injected. Methods that include...

2. The method according to claim 1, wherein the editing includes replacing a subset of tokens in the source NL prompt with one or more substitution tokens that are different from the subset of tokens in the source NL prompt.

3. The method according to claim 2, wherein the one or more features generated based on the editing of the source NL prompt include a text embedding of a modified prompt, the text embedding of a modified prompt conforming to the source NL prompt but replacing the subset of tokens of the source NL prompt with the edited tokens.

4. In at least some of the iterations of generating the edited image using the LLI model, injecting at least a portion of the source cross-attention map is: The method according to claim 3, comprising using the entire source cross-attention map when processing the text embedding of the modified prompt in at least some of the iterations.

5. The method according to claim 4, wherein at least some of the iterations are a subset of the iterations, and in other iterations not included in the subset of the iterations, other cross-attention maps are used when processing the text embedding, and the source cross-attention map is not used when processing the text embedding.

6. The method according to claim 5, wherein the subset of the iterations is an initial consecutive sequence of the iterations.

7. The method according to claim 5, wherein the subset of the iterations comprises more than 5 percent of the iterations and less than 95 percent of the iterations.

8. The method according to claim 5, wherein the subset of the iterations comprises more than 10 percent of the iterations and less than 90 percent of the iterations.

9. The method according to claim 1, wherein the editing includes adding one or more additional tokens to the source NL prompt.

10. The method according to claim 9, wherein the one or more features generated based on the editing of the source NL prompt include text embedding of the source NL prompt and the modified prompt including the additional token.

11. In at least some of the iterations of generating the edited image using the LLI model, injecting at least a portion of the source cross-attention map is: The method according to claim 10, comprising using the entire source cross-attention map when processing a portion of the text embedding corresponding to the source NL prompt, wherein the source cross-attention map is not used when processing additional portions of the text embedding corresponding to the additional token.

12. The method according to claim 11, wherein at least some of the iterations are a subset of the iterations, and in other iterations not included in the subset, the source cross attention map is not used when processing the portion of the text embedding corresponding to the source NL prompt.

13. The method according to claim 12, wherein the subset of the iterations is an initial consecutive sequence of the iterations.

14. The method according to claim 12, wherein the subset of the iterations comprises more than 5 percent of the iterations and less than 95 percent of the iterations.

15. The method according to claim 12, wherein the subset of the iterations comprises more than 20 percent of the iterations and less than 75 percent of the iterations.

16. The method according to claim 1, wherein the editing comprises adjusting emphasis in one or more emphasis tokens among the source tokens of the source NL prompt, the adjustment of emphasis being an increase or decrease in emphasis.

17. The one or more features generated based on the editing of the source NL prompt include one or more scaled attention maps for the one or more emphasis tokens, and further Identifying the emphasis portion of the source cross-attention map corresponding to one or more emphasis tokens, The method according to claim 16, comprising generating one or more scaled attention maps by scaling the emphasized portion in accordance with the adjustment of the emphasis.

18. The adjustment of emphasis is an increase in emphasis, and generating one or more scaled attention maps by scaling the emphasis portion in response to the adjustment of emphasis is, The method according to claim 17, comprising increasing the value of the emphasized portion by a coefficient.

19. The method according to claim 18, wherein the increase in emphasis indicated by the user interface input is of a specific magnitude, which is one of a plurality of candidate magnitudes, and the coefficient is proportional to the specific magnitude.

20. The method according to claim 17, wherein the text embedding of the source NL prompt is processed in the iteration of processing using the LLI model, and the text embedding includes an emphasis embedding portion corresponding to one or more emphasis tokens and a rest embedding portion corresponding to the rest of the source NL prompt after the emphasis portion has been removed.

21. The at least portion of the source cross attention map is the remaining portion of the source cross attention map after the emphasis portion has been removed, and in at least some of the iterations of generating the edited image using the LLI model, the at least portion of the source cross attention map is injected. The method according to claim 20, comprising using the remaining portion of the source cross attention map when processing the embedding of the remaining portion in at least some of the iterations, wherein one or more scaled source cross attention maps are used when processing the enhanced embedding portion in at least some of the iterations.

22. The method according to claim 16, wherein at least some of the iterations are all of the iterations.

23. The method according to claim 1, wherein the source cross attention map includes values ​​that associate tokens of the source NL prompt with pixels of the source image.

24. The method according to claim 23, wherein each of the aforementioned values ​​defines the corresponding weight of the corresponding token in the corresponding pixel of the aforementioned pixel.

25. The source image is generated based on processing the source natural language (NL) prompt using the LLI model. The method according to claim 1, further comprising:

26. The method according to claim 1, wherein the user interface input indicating the edit to the source NL prompt includes interaction with a graphical user interface that renders the typed input and / or the source NL prompt.

27. The editing includes adjusting the emphasis in one or more emphasis tokens among the source tokens of the source NL prompt, wherein the adjustment of emphasis is an increase or decrease in emphasis. The method according to claim 26, wherein the user interface input includes the interaction with the graphical user interface, and the interaction includes the interaction with a slider corresponding to one or more emphasis tokens.

28. The user interface input indicating the editing of the source NL prompt includes speech input captured in audio data, and further, Using an automatic speech recognition model, the audio data is processed to generate recognized text corresponding to the speech input, The method according to claim 1, comprising processing the recognized text to determine the edit to the source NL prompt.

29. A method implemented by one or more processors, wherein the method is Identifying real images captured by a real camera, Identifying natural language (NL) captions for the aforementioned real images, Using an inversion process and based on the real image, generate a noise vector for the real image, Using a Large-Scale Language Image (LLI) model and the noise vector, the NL caption is processed to generate a source image that approximates the real image. When generating the source image, the source cross-attention map generated using the cross-attention layer of the LLI model is identified, Identifying the source random seed used when generating the aforementioned source image, Following the generation of the aforementioned source image, The system receives user interface input indicating edits to the NL caption used when generating the source image, In response to receiving the user interface input indicating the edit to the NL caption, The process includes, in multiple iterations of the process using the LLI model, generating an edited image that is visually similar to the source image but includes visual modifications that match the edits to the NL caption indicated by the user interface input, and in the multiple iterations of the process using the LLI model, generating the edited image is In the iteration of the process using the LLI model, One or more features generated based on the editing of the NL caption, The aforementioned source random seed and the process of, In at least some of the iterations of generating the edited image using the LLI model, at least a portion of the source cross-attention map is injected. Methods that include...

30. The method according to claim 29, wherein the NL caption for the real image is generated based on other user interface inputs.

31. The method according to claim 29, wherein the NL caption for the real image is automatically generated based on processing the real image using an additional model trained to predict captions for images.

32. The method according to claim 29, wherein the inversion process includes using a deterministic denoising diffuse implicit model (DDIM).

33. A system comprising: a memory for storing instructions; and one or more processors operable to execute the instructions in order to carry out the method according to any one of claims 1 to 32.

34. One or more temporary or non-temporary computer-readable media for storing instructions, wherein, when executed by one or more processors, the instructions cause the method according to any one of claims 1 to 32 to be carried out.

Images