Text-guided image editor

Abstract

To provide an image editor that makes a semantic change to a given image on the basis of a text instruction.SOLUTION: A method comprises the steps of: converting a base prompt and an edit prompt to base embedding and edit embedding; determining new edit embedding on the basis of a time step associated with a plurality of iteration processes and a time-step dependent weight that controls blending of the base embedding and the edit embedding; inputting the base embedding into a diffusion model in a base reverse direction process configured to update a base latent image associated with the base image; inputting the new edit embedding into the diffusion model in an edit reverse direction process that updates an edit latent image associated with the edited image, and inputting a cross-attention map generated from the diffusion model in the base reverse direction process into the diffusion model in the edit reverse direction process; and converting the edit latent image into an edited image and outputting the edited image.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] FIELD OF THE INVENTION Embodiments of the invention described herein relate to text-guided image editing, and in particular to computer-implemented methods, computer programs, and information programming apparatus. [Background technology]

[0002] Text-guided image editing refers to methods based on image generation models that make semantic changes to a given image based on textual instructions. Text-guided image editing involves using textual descriptions to guide the modification or manipulation of an image. This can be achieved through techniques such as conditional image generation, image captioning, semantic image editing, conversational interfaces, and content-aware editing. It enables an intuitive editing workflow, allows complex instructions using natural language, and finds applications in graphic design, photo editing, content creation, and computer-aided design.

[0003] It is desirable to be able to control image editing and preserve the identity of the subject of the image. Summary of the Invention [Problem to be solved by the invention]

[0004] It is an object of the present disclosure to at least partially address one or more of the above-mentioned problems. The invention is defined in the independent claims, to which reference should now be made. Further features are set out in the dependent claims. [Means for solving the problem]

[0005] According to one embodiment, a computer-implemented method is disclosed herein that includes obtaining a base prompt indicating a base image and an edit prompt indicating edits to be made to the base image. The method further includes converting the base prompt and the edit prompt into a base embedding and an edit embedding, respectively. The method further includes repeating, over multiple iterations, (i) determining a new edit embedding based on the base embedding and the edit embedding, a time step associated with the iteration, and a weight dependent on the time step, the weight controlling blending of the base embedding and the edit embedding; (ii) inputting the base embedding into a diffusion model in a base backward process configured to update a base latent image associated with the base image; and (iii) inputting the new edit embedding into a diffusion model in an edit backward process configured to update an edit latent image associated with an edited image, the cross-attention map generated from the diffusion model in the base backward process being input into the diffusion model in the edit backward process. The method further includes converting the edit latent image into an edited image, and finally outputting the edited image. [Brief explanation of the drawings]

[0006] Embodiments of the present invention will now be further described, by way of example only, with reference to the accompanying drawings, in which like reference numerals refer to like parts and in which: [Figure 1] FIG. 1 illustrates a stable diffusion model. [Figure 2] FIG. 10 is a diagram illustrating a cross-attention mechanism. [Figure 3] FIG. 10 is a diagram illustrating a comparison method. [Figure 4] FIG. 10 is a diagram illustrating a comparison method. [Figure 5] FIG. 10 is a diagram illustrating a comparison method. [Figure 6] FIG. 1 illustrates an embodiment of the present invention. [Figure 7] FIG. 1 illustrates an embodiment of the present invention. [Figure 8] An example showing a comparison between a comparative method (top row) and an embodiment of the present invention (bottom row) is shown. [Figure 9] An example showing a comparison between a comparative method (top row) and an embodiment of the present invention (bottom row) is shown. [Figure 10] FIG. 1 illustrates an embodiment of the present invention. [Figure 11] FIG. 1 illustrates an embodiment of the present invention. [Figure 12] 1 is a flowchart illustrating an embodiment of the present invention. [Figure 13] FIG. 1 shows an apparatus. [Figure 14] 1 is a flowchart illustrating an embodiment of the present invention. [Figure 15] 1 shows examples of a base image and an edited image according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0007] Text-guided image editing refers to a method based on an image generation model that makes semantic changes to a given image based on text instructions. The present disclosure aims to provide a text-guided image editor that provides improved control over the editing process (e.g., controlling the amount of smile when editing a facial expression to smile) and better preserves target identity (e.g., preserving the subject's facial features when editing a facial expression). The present disclosure also aims to provide a text-guided image editor that does not require training for each new edit, unlike existing methods that require training for each edit. The present disclosure provides these advantages by providing an embedded mixer that controls the intertwining of base prompts (prompts about the original image, e.g., "photo of a man") and edit prompts (prompts about the edited image, e.g., "photo of a man smiling").

[0008] Figure 1 illustrates a stable diffusion model 100. The diffusion model is a state-of-the-art generative model that synthesizes images from white Gaussian noise through progressive denoising using a T-step dediffusion process. In addition to high-quality image generation, the diffusion model has proven useful for text-guided semantic image editing, e.g., changing facial expressions in images. Some methods perform text-guided image editing by fine-tuning a pre-trained unconditional diffusion model using a contrastive language image pretraining (CLIP)-based loss. However, these methods are computationally expensive due to the required fine-tuning. Stable diffusion is an open-source conditional text-image diffusion model that uses a cross-attention mechanism to condition the dediffusion process with a CLIP embedding E of the text prompt. Referring to the figure, a text prompt 102 (e.g., "photo of a person") is input to a CLIP text encoder 104 (via a tokenizer, not shown), which outputs a text embedding E 106. The text embeddings 106 are fed to the diffusion model 108 at each time step of the inverse process through a cross-attention mechanism 112. Specifically, the diffusion model in stable diffusion consists of 16 cross-attention layers. The i-th cross-attention layer at the t-th diffusion step is fed to the input feature F t i and extracts those features by conditioning on text embeddings.

number

[0009] Figure 2 illustrates a cross-attention mechanism 200. Cross-attention mechanisms enable neural network models to selectively focus on relevant portions of one sequence based on information provided by another sequence. Cross-attention mechanisms typically operate with three sets of input sequences: a "query" sequence, a "key" sequence, and a "value" sequence. These sequences can represent various types of data, such as words in a sentence, tokens in a document, or pixels in an image.

[0010] Referring to the figure, a base prompt 202 is input to a CLIP text encoder 204 (via a tokenizer, not shown), which outputs a text embedding E 206. The text embedding 206 is used to obtain a key-value input sequence, while the input features F i t 208 is used to obtain a query input sequence. Both inputs 208, 206 are input to a cross-attention mechanism 210. Each element in the query sequence is associated with a "query vector" 212, while each element in the key-value sequence is associated with both a "key vector" 214 and a "value vector" 216. These vectors are used to represent the semantic information of the input sequence. Specifically, the cross-attention mechanism 210 uses W to obtain the key and value sequences, respectively. K and W V The input features 206 also include a linear projection of the text embedding using W Q The query sequence is then linearly projected using the attention weights, which are then calculated using a similarity measure, such as the dot product or scaled dot product, between the query vector and the key vector.

[0011] The attention weights can be visualized as a cross-attention map 218. These maps typically show a grid where each row corresponds to an element in the query sequence and each column corresponds to an element in the key-value sequence. The intensity or color of each cell in the grid represents the amount of attention assigned to the corresponding pair of elements. The attention weights are calculated by computing a weighted sum of the value vectors in the key-value sequence to produce the final output feature ^F i t 220. This weighted sum represents the "interesting" information from the value-sequence that is relevant to each element in the query sequence. During training, W Q , W K , W V The parameters of the cross-attention mechanism, including the weights, are learned through backpropagation using labeled data or other suitable training objectives. The cross-attention mechanism allows the model to effectively leverage contextual information from one sequence to improve the processing of another sequence.

[0012] FIG. 3 illustrates a first comparative method 300 (Comparative Method 1). Comparative Method 1 is a method for editing an image using text prompts 302. This method fine-tunes an unconditional diffusion model for the desired edits by minimizing a CLIP-based loss, which is calculated as the dissimilarity between the CLIP text embedding of the edit prompts 302 and the CLIP image embedding of the edited image. First, a forward denoising diffusion implicit model (DDIM) process 308 is used to decode the base image 304 into a corresponding latent image x T 306. Then, an inverse DDIM process 312 inverts the latent image x TThe diffusion model 310 is fine-tuned so that when applied to 306 it gives the desired result (edited image 314). The problem with comparative method 1 is that separate diffusion models need to be trained for different types of edits, which is time-consuming and memory-intensive. The first comparative method may be referred to as DiffusionCLIP (Non-Patent Document 1). [Non-Patent Document 1] Kim et al. 2022, “DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation”

[0013] 4 and 5 show a second comparative example method 400, 500 (Comparative Example Method 2). Comparative Example Method 2 is a method for editing images using text prompts that uses a cross-attention map to control edits. Unlike Comparative Example Method 1, Comparative Example Method 2 does not require training for individual edits; i.e., the model does not need to be fine-tuned for different edits.

[0014] As shown in FIG. 4 , in comparative example Method 2, two backward processes, namely, a base backward process 402 for the base prompt and an edit backward process 404 for the edit prompt, are performed simultaneously. During the process, for each step t of the overlapping portion of the text prompt, the edit backward process 404 uses the cross-attention map generated by the cross-attention processor 406 from the base backward process 402. For example, base prompt: Photo of man, edit prompt: Photo of smiling man. In the edit backward process, the cross-attention maps for the words "Photo," "of," and "man" are taken from the base prompt, while the map for the word "smiling" is calculated in the edit backward process itself. More specifically, a latent image 408 is input to both the base backward process 402 and the edit backward process 404. The base backward process 402 generates the base embedding E base 404 to generate a denoised base latent image 410. At the same time, the redaction backward process 404 edit The denoised edited latent image 412 is generated using the denoised edited latent image 412. The cross-attention map generated from the base backward process 402 is input to the edit backward process 404. The denoised base latent image 410 is input to a latent-to-image decoder 414 to generate a base image 416. The denoised edited latent image 412 is input to a latent-to-image decoder 414 to generate an edited image 418.

[0015] Figure 5 shows the cross-attention mechanism for comparative example Method 2. The base cross-attention process 502 functions similarly to the cross-attention mechanism described above in connection with Figure 2. The edit cross-attention process 504 functions similarly to the base cross-attention process 502, except for the additional input of a base cross-attention map 506, which is generated from the base backward process and input to the edit backward process.

[0016] The problem with method 2 in the comparative example is that it has problems with changing the identity of the subject. Context encoding by the text encoder allows editing instructions to affect other word embeddings from editing prompts. For example, in the above case, the embedding for the word "man" is E base and E edit The difference is in the editing intensity. When the word "smiling" changes, its embedding changes due to the context embedding by the CLIP text encoder. This may have a negative impact on the subject's identity. A further problem with comparative method 2 is that it is difficult to control the editing intensity. Comparative method 2 proposes to control the editing intensity by scaling the cross-attention map corresponding to the edited phrase (e.g., scaling the cross-attention map of the word "smiling" in the example above). However, this is not very effective. Now, E edit This is because other words in also encode the "Smiling" attribute due to context encoding. The method of the second comparative example may be referred to as Prompt-to-Prompt. [Non-patent document 2] Hertz et al. 2022, “Prompt-to-Prompt Image Editing with Cross-Attention Control”

[0017] 14 is a flowchart illustrating method steps in a computer-implemented method 1400 for generating an edited image, according to some embodiments of the present invention. In step 1402, a base prompt indicating a base image is obtained. The base prompt may be obtained from an existing image and / or user input. The base prompt may include a text description of the base image. In step 1404, an edit prompt indicating edits to be made to the base image is obtained. The edit prompt may be obtained from user input. The edit prompt may include a text description of the edited image. In step 1406, the base prompt and the edit prompt are each obtained from a base image. base and Edit E edit In step 1408, the new edit embedding ^E edit (t) is the base and edit embedding, the time step t and the weights w e (t), where the weights control the blending of the base and edit embeddings. For example, the weights may depend on the time step, such that in earlier time steps, the base and edit embeddings blend less than in later time steps. The weights may further depend on a time-invariant parameter that controls how much the base and edit embeddings blend. This time-invariant parameter is referred to herein as the hyperparameter σ eIn step 1410, the base embedding is input to a diffusion model in a base backward process configured to update a base latent image associated with the base image. In step 1412, the new edit embedding is input to a diffusion model in an edit backward process configured to update an edit latent image associated with the edited image, and a cross-attention map generated from the diffusion model in the base backward process is input to the diffusion model in the edit backward process. Steps 1410 and 1412 may overlap in time, e.g., they may occur simultaneously. In step 1414, the edit latent image is converted into an edited image. In step 1416, the edited image is output, e.g., to a user. This may be in the form of displaying the edited image to the user via, e.g., a graphical user interface (GUI).

[0018] Embodiments of the present invention may work with images of any subject. For example, the base image may include an image of a human face. In such an example, editing may include changing the facial expression of the human face. In other examples, editing may include changing the facial features and / or age of the human face. Examples of editing images of human faces are shown in FIGS. 8 and 9 and described below. An example 1500 of editing an image of an animal is shown in FIG. 15, which shows an image of a bird 1502 edited to show an image of a flying bird 1504. FIG. 15 also shows an image of a horse 1506 edited to show an image of a running horse 1508. FIG. 15 shows an image of a dog 1510 edited to show an image of a running dog 1512. Thus, editing can include changing what the subject of the image is doing. Editing can include changes to the background of the image.

[0019] FIG. 6 illustrates an embodiment 600 of the present invention, and in particular, the t-th diffusion step of the method of the present disclosure. This method is similar to the Prompt-to-Prompt method discussed in connection with FIGS. 4 and 5, but introduces an embedded mixer module. The embedded mixer addresses the issue of an edited word (e.g., "smiling") affecting the embedding of other words (e.g., "man") in the edit prompt. Referring to the figure, method 600 includes inputting a base prompt 602 (e.g., "photo of a man") and an edited prompt 604 (e.g., "photo of a smiling man") to a CLIP text encoder 606 (via a tokenizer, not shown). The CLIP text encoder generates a base embedding E base and edit embed E edit The base embedding E base and edit embed E edit is input to the embedded mixer 608 along with the time step t since the embedded mixer output depends on the time step. The embedded mixer 608 then outputs the edit embedding ^E which depends on the new time step t. edit Output (t). Base embedding E base is also the base latent image L t+1 base Acting on L t base (i.e., the original unedited image in latent space) is input to the diffusion model in the base inverse process 610, which generates the new edit embedding ^E edit (t) is the edited latent image L t+1 edit Acting on L t editThe cross-attention map from the base backward process 610 is input to the diffusion model in the edit backward process 612, which generates the edited image (i.e., the edited image in latent space). The cross-attention processor 614, as described above in connection with Figures 2 and 5, acts between the two backward processes 610, 612, allowing the process that obtains the original unedited image to inform the process that obtains the edited image. The cross-attention processor 614 uses the cross-attention map from the base backward process 610 to generate the edited image.

number

number

number

[0020] 7 shows the embedded mixer 608 in more detail (700). Referring to the figure, a base prompt, e.g., "photo of a man" 602, and an edit prompt, e.g., "photo of a smiling man" 604, are input to a tokenizer 702, which generates the base token t b and edit token t e These tokens are then input to a text encoder 606, which generates the base embedding E base 704 and Edit Embed E edit The embedding of the edited word "smiling" is shown as 706. The embeddings 704 and 706 are input to the embedding mixer 608, as described above with reference to Figure 6. The mix is eThis is done through a convex combination of the embeddings of the base and overlapping portions of the text prompt using (t). These weights are designed to emphasize the base embedding for early steps in the dediffusion process and the edit embedding for later parts of the process. Because early time steps in the dediffusion process affect higher-level features of the image (e.g., facial structure if the image is of a face), this ensures that only the base embedding is used to influence these higher-level features to maintain the subject's identity. Later time steps in the dediffusion process affect lower-level features (i.e., finer details) of the image (e.g., facial expression if the image is of a face). By including the edit embedding later in the process, this ensures that the edit embedding only affects the finer details of the image and does not completely change the subject of the image. An embedding specifying the desired edit is then concatenated with the blended embedding. Because the base embedding does not intertwine with the edit embedding, problems related to entanglement are mitigated. Because the context within the edit prompt is also important, blending is performed instead of replacing embeddings.

[0021] New Edit Embed ^E edit (t) may be expressed as follows:

number

number

[0022] FIG. 8 shows an exemplary comparison 800 of Prompt-to-Prompt results 804-808 (top row) with methods 810-814 (bottom row) of the present disclosure. For both methods, an original image 802 created using the base prompt "Photo of a Man" is edited with the editing prompts "Photo of a Smiling Man," "Photo of a Crying Man," and "Photo of an Old Man." The results of the Prompt-to-Prompt method for editing the "Photo of a Smiling Man" can be seen in image 804. The results of the Prompt-to-Prompt method for editing the "Photo of a Crying Man" can be seen in image 806. The results of the Prompt-to-Prompt method for editing the "Photo of an Old Man" can be seen in image 808. The results of the disclosed method for editing the "Photo of a Smiling Man" can be seen in image 810. The results of the disclosed method for editing the "Photo of a Crying Man" can be seen in image 812. The results of the disclosed method for editing "old man photo" can be seen in image 814. For the disclosed method, all three of these results are consistent with the hyperparameter value σ e = 500. These results demonstrate that the results of the prompt-to-prompt method can alter the identity of the subject. For example, the result for "Photo of smiling man" 804 appears older than the base image 802. The "smiling" embedding unintentionally affected the subject's age. Similarly, the result for "Photo of crying man" 806 has a different facial structure than the base image 802. The "crying" embedding unintentionally affected the subject's facial structure. In contrast, the results 810, 812, and 814 of the disclosed method have much more realistic results, and it is clear that the people in the edited images 810, 812, and 814 are the same as those in the base image 802.

[0023] Figure 9 shows an example comparison 900 of prompt-to-prompt results 902-908 (top row) with methods 910-916 of the present disclosure (bottom row). In the top row, images 902-908 show a test of controllability in the prompt-to-prompt approach by varying the weights in the cross-attention map corresponding to the word "old." Image 902 shows the w a = 0.1, and image 904 has w a =0.3, image 906 is w a =0.7, image 908 is w a = 1.0. This approach was proposed in the Prompt-to-Prompt paper. As can be seen in images 902-908, this is not a robust way to control the degree of editing (e.g., to control how old a man looks), and can fail for some images as shown. In contrast, in the bottom row, images 910-916 show a good fit with the hyperparameter σ, which controls the blending of the embeddings. e 9 illustrates a test of controllability for the method of the present disclosure by varying σ As can be seen in images 910-916, the editing strength can be effectively controlled using this method, with lower hyperparameter values ​​(e.g., 100) resulting in stronger editing (e.g., a man looking older) and higher hyperparameter values ​​(e.g., 700) resulting in weaker editing (e.g., a man looking younger). The example hyperparameter values ​​shown in FIG. 9 are σ for image 910, σ for image 912, and σ for image 916. e = 100, and for image 912, σ e = 300, and for image 914, σ e = 500, and for image 916, σ e =700.

[0024] 10 illustrates an embodiment of the present invention in more detail (1000). The edit prompt 604 and the base prompt 602 are input to a tokenizer 702, which generates an edit token t e and base token t bThese tokens are then input to a text encoder 606, which generates the base embedding E base and edit embed E edit The mask and alignment index vector 1002 is a component used to align the tokens of the base prompt and edit prompt. b , t e It takes as input m and outputs i, where i is the number of b The same token in t e is the index vector to align with t b and t e m is calculated by comparing t b [i] and t e is a binary mask that takes a value of 1 for the same token in the embedding E (as defined above in relation to Figure 7). base , E edit are input to the embedded mixer 608 as described above with reference to Figures 6 and 7, along with the time steps t, m, and i. As described above, the embedded mixer 608 generates a new, time step t-dependent edit embedding ^E edit The diffusion models 610, 612 and the cross-attention processor 614 are described above in connection with FIG.

[0025] Figure 11 illustrates an embodiment of the present invention in more detail (1100). Figure 11 shows how the tokenizer 702, mask and alignment index vector 1002, and text encoder 606 stages (described above) are calculated once at the beginning of the backward process. The embedded mixer 608, diffusion models 610, 612, and cross attention processor 614 stages are calculated once at the beginning of the backward process.

number

[0026] 12 is a flowchart 1200 illustrating an embodiment of the present invention. At step 1202, the method starts. At step 1204, the method checks whether there is an actual base image to be edited. Embodiments of the present invention can work from the actual image and work backward to generate appropriate prompts that give the actual image, or the user can provide prompts to generate the base image. If there is an actual image, the method proceeds to step 1206, where the actual image is loaded and a latent image L is calculated using the null text inversion method, where t is set to t=T, and the base latent image L is calculated. b t is L b t =L e t Alternatively, if there is no real image, the method proceeds to step 1208, where the method sets t=T and the latent image L b t =L e t .about.N(0,1). After step 1206 or step 1208, the method proceeds to step 1210, where the base prompt 602 and edit prompt 604 are loaded with a sample initial latent vector. The sample initial latent vector is the latent vector L T , i.e., the starting point of the de-diffusion process. This is the same for both the base process and the reverse process. An editing prompt 604 may be provided by the user. In step 1212, both the base prompt and the editing prompt are tokenized by the tokenizer 702 to produce the base token t b and edit token t e In step 1214, a mask vector m and an index vector i are calculated according to the mask and alignment index vector 1002. In step 1216, the base E base and Edit E edit The embedding is computed by the text encoder 606. In step 1218, the modified edit embedding ^E edit is calculated in the embedded mixer 608. In step 1220, the latent image Lb t and L e t is updated by the base and edit backward process of the diffusion models 610, 612 and the cross attention processor 614. b t-1 and L e t-1 In step 1222, the method checks whether time step t>1 (i.e., whether this is the last time step of the process). If t>1 (i.e., it is not the last time step), the method proceeds to step 1224, where t←t-1 (i.e., changes t to the next time step), and then repeats steps 1218-1222. If t<1 (i.e., it is the last time step), the method proceeds to step 1226, where the edited latent image L e 0 is decoded to obtain the edited image. At step 1228, the method ends.

[0027] The method disclosed herein provides an improved text-guided image editing method by providing an embedded mixer for better control of the editing process.

[0028] Text-guided image editing may be used for any number of downstream tasks. Text-guided image editing is advantageous because it provides a user-friendly, intuitive way to create or modify images without requiring specialized knowledge of graphic design or image editing software. Users can input text descriptions instead of manually manipulating image elements, making it accessible to a wider range of individuals. Text-to-image editing can be faster than traditional methods, especially for generating multiple variations of an image. Users can quickly describe desired changes, and the software can generate or edit the image accordingly, saving time and effort. By generating images based on text descriptions, text-to-image editing can automate repetitive tasks and streamline workflow. Additionally, users can specify precise details or preferences in their text input, enabling customization.

[0029] 13 is a block diagram of an information processing apparatus 1300 or computing device 1300, such as a data storage server, that may be used to embody the present invention, to implement some or all of the operations of the methods embodying the present invention, and to perform some or all of the tasks of the apparatus of an embodiment. The computing device 1300 may be used to perform any of the method steps described above, e.g., any of steps S.1402 through S.1416, and / or any of the processes described above.

[0030] Computing device 1300 includes a processor 1302 and memory 1304. Optionally, the computing device also includes a network interface 1306 for communicating with other such computing devices. Optionally, the computing device also includes one or more input mechanisms, such as a keyboard and mouse 1308, and one or more display units, such as a monitor or monitors 1310. These elements can facilitate user interaction. The components can be connected to each other via a bus 1312.

[0031] Memory 1304 may include a computer-readable medium, which term may refer to a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) configured to carry computer-executable instructions. Computer-executable instructions may include, for example, instructions and data that are accessible by a computer (e.g., one or more processors) and cause the computer to perform one or more functions or operations. For example, computer-executable instructions may include instructions for implementing a method disclosed herein or any method step(s) disclosed herein, e.g., any of steps S.1402-S.1416, and / or any process described above. Thus, the term "computer-readable storage medium" may also include any medium capable of storing, encoding, or carrying a set of instructions for execution by a machine and causing a machine to perform any one or more of the method steps of the present disclosure. Thus, the term "computer-readable storage medium" may be interpreted to include, but is not limited to, solid-state memory, optical media, and magnetic media. By way of example and not limitation, such computer-readable media may include non-transitory computer-readable storage media including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory devices (e.g., solid-state memory devices).

[0032] The processor 1302 is configured to control a computing device and perform processing operations, for example, executing computer program code stored in the memory 1304 to perform any of the method steps described herein. The memory 1304 stores data read and written by the processor 1302 and can store the data and / or programs for performing any of the method steps and / or processes described above. As referred to herein, a processor may include one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like. The processor may include a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or combinations of instruction sets. The processor may also include one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. In one or more embodiments, the processor is configured to execute instructions for performing the operations described herein. The processor 1302 may be considered to include any of the modules described above. Any operation described as being implemented by a module may be implemented by a computer, for example by the processor 1302, as a method.

[0033] The memory 1304 and the processor 1302 may be collectively configured to provide an embedding mixer 608 configured to perform step S.1408 of determining new edit embeddings. The memory 1304 and the processor 1302 may be collectively configured to provide a diffusion model. The memory 1304 and the processor 1302 may be collectively configured to provide a cross-attention processor 614 configured to generate a cross-attention map. The memory 1304 and the processor 1302 may be collectively configured to provide a tokenizer 702 configured to convert base and edit prompts into base and edit tokens, and a text encoder 606 configured to convert base and edit tokens into base and edit embeddings. The memory 1304 and the processor 1302 may be collectively configured to provide a mask and alignment index vector module 1002 configured to calculate mask vectors and index vectors based on the base tokens and edit tokens.

[0034] The display unit 1310 may display representations of data stored by the computing device to allow a user to interact with the apparatus 1300, for example, by drag-and-drop or selection interactions, and / or any other output described above, and may also display cursors and dialog boxes and screens that enable interaction between a user and programs and data stored on the computing device. The input mechanism 1308 may allow a user to input data and instructions into the computing device, such as allowing a user to enter any user input described above.

[0035] The network interface (network I / F) 1306 may be connected to a network such as the Internet and may be connected to other such computing devices via the network. The network I / F 1306 controls data input and output to and from other devices via the network. Other peripheral devices such as a microphone, speakers, printer, power supply unit, fan, case, scanner, trackball, etc. may also be included in the computing device.

[0036] A method embodying the present invention may be performed on a computing device / apparatus 1300 such as that shown in FIG. 13. Such a computing device need not have all of the components shown in FIG. 13, but may instead consist of a subset of those components. For example, apparatus 1300 may include a processor 1302 and a memory 1304 connected to the processor 1302. Alternatively, apparatus 1300 may include a processor 1302, a memory 1304 connected to the processor 1302, and a display 1310. A method embodying the present invention may be performed by a single computing device that communicates with one or more data storage servers over a network. The computing device may also be a data storage device itself that stores at least a portion of the data.

[0037] Methods embodying the present invention may be performed by a plurality of computing devices operating in cooperation with one another, one or more of which may be data storage servers that store at least a portion of the data.

[0038] The invention may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The invention can also be implemented as a computer program or computer program product, i.e., a computer program tangibly embodied in a non-transitory information carrier, for example a machine-readable storage device or a propagated signal, for execution by, or to control the operation of, one or more hardware modules.

[0039] A computer program may be in the form of a stand-alone program, a computer program portion, or two or more computer programs, may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a data processing environment. A computer program may be deployed to be executed on one module or on multiple modules at one site, or may be distributed across multiple sites and interconnected by a communications network.

[0040] The method steps of the present invention may be performed by one or more programmable processors executing a computer program that performs the functions of the present invention by operating on input data and generating output. The apparatus of the present invention may be implemented as programmed hardware or as special purpose logic circuitry including, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0041] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a processor for executing instructions coupled to one or more memory devices for storing instructions and data.

[0042] The above-described embodiments of the present invention may be advantageously used independently of other embodiments or in any feasible combination with one or more other embodiments. While certain embodiments have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. Indeed, the novel methods and apparatus described herein may be embodied in a variety of other forms, and various omissions, substitutions, and modifications may be made in the methods and apparatus described herein.

[0043] A list of numbered statements. The following additional notes are provided regarding the embodiments including the above examples. (Appendix 1) 1. A computer-implemented method for image editing, the method comprising: obtaining a base prompt indicating a base image and an edit prompt indicating an edit to be made to the base image; converting the base prompt and edit prompt into a base embedding and an edit embedding, respectively; Over multiple iterations, determining a new edit embedding based on the base embedding and the edit embedding, a time step associated with the iteration, and weights that depend on the time step, the weights controlling the blending of the base embedding and the edit embedding; inputting the base embedding into a diffusion model in a base backward process configured to update a base latent image associated with the base image; and inputting the new edit embeddings into the diffusion model in an edit backward process configured to update an edited latent image associated with the edited image, wherein a cross-attention map generated from the diffusion model in the base backward process is input into the diffusion model in the edit backward process. Repeating and; converting the edited latent image to the edited image; and outputting the edited image. method. (Appendix 2) 2. The method of claim 1, wherein the weights depend on the time step such that at earlier time steps the base embedding and the edit embedding blend less than at later time steps. (Appendix 3) 3. The method of claim 1 or 2, wherein the weights further depend on a time-invariant parameter that controls how much the base embedding and the edit embedding blend. (Appendix 4) 4. The method of any one of claims 1 to 3, wherein converting the base prompt and edit prompt into base embedding and edit embedding includes converting the base prompt and edit prompt into base tokens and edit tokens, and converting the base tokens and edit tokens into base embedding and edit embedding. (Appendix 5) 5. The method of claim 4, wherein the new edit embedding is further based on a mask vector and an index vector calculated based on the base token and edit token. (Appendix 6) 6. The method of any one of claims 1 to 5, wherein inputting the base embedding into the diffusion model in the base backward process and inputting the new edit embedding into the diffusion model in the edit backward process overlap in time. (Appendix 7) 7. The method of any one of claims 1 to 6, wherein the base prompt is derived from an image. (Appendix 8) 8. The method of any one of claims 1 to 7, wherein the base prompt and / or the editing prompt are obtained from user input. (Appendix 9) 9. The method of any one of claims 1 to 8, wherein inputting the base embedding into the diffusion model in the base backward process and inputting the new edit embedding into the diffusion model in the edit backward process are performed simultaneously. (Appendix 10) 10. The method of any one of claims 1 to 9, wherein the base prompt includes a text description of the base image. (Appendix 11) 11. The method of any one of claims 1 to 10, wherein the editing prompt includes a text description of the edited image. (Appendix 12) 12. The method of any one of claims 1 to 11, wherein the base image comprises an image of a human face. (Appendix 13) 13. The method of claim 12, wherein the editing includes a change in facial expression of the person. (Appendix 14) 14. The method of any one of claims 1 to 13, wherein the base embedding and edit embedding comprise vector representations of the base prompt and edit prompt, respectively. (Appendix 15) 15. The method of any one of claims 1 to 14, wherein the step of obtaining the base prompts and edit prompts and converting them into embeddings is performed only once per edit. (Appendix 16) 16. The method of any one of claims 1 to 15, wherein the weights depend on the time step such that the base embedding has more weight at earlier time steps than at later time steps. (Appendix 17) 17. The method of any one of claims 1 to 16, wherein converting the base prompts and edit prompts into the base embeddings and edit embeddings comprises converting the base prompts and edit prompts into base tokens and edit tokens via a tokenizer unit, and converting the base tokens and edit tokens into base embeddings and edit embeddings via a text encoder unit. (Appendix 18) 18. The method of claim 17, wherein the text encoder unit includes a contrastive language image pre-training (CLIP) text encoder. (Appendix 19) 19. The method of any one of claims 1 to 18, wherein the edited latent image is converted into the edited image using a latent image-to-image decoder. (Appendix 20) 20. The method of any one of claims 1 to 19, wherein the mask vector and the index vector are calculated by a mask and alignment vector unit. (Appendix 21) 21. The method of any one of claims 1 to 20, wherein the new edit embedding is determined by an embedding mixer unit that receives the base embedding and edit embedding, the time step, and the weights as inputs and outputs the new edit embedding. (Appendix 22) 22. The method of any one of claims 1 to 21, wherein the cross-attention map is generated by a cross-attention processor. (Appendix 23) A computer program product that, when executed on a computer, causes the computer to perform a method, the method comprising: obtaining a base prompt indicating a base image and an edit prompt indicating an edit to be made to the base image; converting the base prompt and edit prompt into a base embedding and an edit embedding, respectively; Over multiple iterations, determining a new edit embedding based on the base embedding and the edit embedding, a time step associated with the iteration, and weights that depend on the time step, the weights controlling the blending of the base embedding and the edit embedding; inputting the base embedding into a diffusion model in a base backward process configured to update a base latent image associated with the base image; and inputting the new edit embeddings into the diffusion model in an edit backward process configured to update an edited latent image associated with the edited image, wherein a cross-attention map generated from the diffusion model in the base backward process is input into the diffusion model in the edit backward process. Repeating and; converting the edited latent image to the edited image; and outputting the edited image. Computer programs. (Appendix 24) 1. An information processing device comprising a memory and a processor coupled to the memory, the processor configured to execute a method, the method comprising: obtaining a base prompt indicating a base image and an edit prompt indicating an edit to be made to the base image; converting the base prompt and edit prompt into a base embedding and an edit embedding, respectively; Over multiple iterations, determining a new edit embedding based on the base embedding and the edit embedding, a time step associated with the iteration, and weights that depend on the time step, the weights controlling the blending of the base embedding and the edit embedding; inputting the base embedding into a diffusion model in a base backward process configured to update a base latent image associated with the base image; and inputting the new edit embeddings into the diffusion model in an edit backward process configured to update an edited latent image associated with the edited image, wherein a cross-attention map generated from the diffusion model in the base backward process is input into the diffusion model in the edit backward process. Repeating and; converting the edited latent image to the edited image; and outputting the edited image. Information processing device. (Appendix 25) 25. The information processing device of claim 24, wherein the memory and the processor are collectively configured to provide an embedding mixer configured to perform the determining of the new edit embedding. (Appendix 26) 26. The information processing device of claim 24 or 25, wherein the memory and the processor are collectively configured to provide the diffusion model. (Appendix 27) 27. The information processing apparatus of any one of claims 24 to 26, wherein the memory and the processor are collectively configured to provide a cross-attention processor configured to generate the cross-attention map. (Appendix 28) Converting the base prompt and edit prompt to a base embedding and an edit embedding includes converting the base prompt and edit prompt to a base token and an edit token, and converting the base token and edit token to a base embedding and an edit embedding, wherein the memory and the processor: a tokenizer configured to convert the base prompt and edit prompt into base tokens and edit tokens; a text encoder configured to convert the base tokens and edit tokens into base embeddings and edit embeddings; 28. The information processing device of any one of clauses 24 to 27, collectively configured to provide: (Appendix 29) 29. The information processing apparatus of any one of claims 24 to 28, wherein the new edit embedding is further based on a mask vector and an index vector calculated based on the base token and edit token, and wherein the memory and the processor are collectively configured to provide a mask and alignment index vector module configured to perform the steps of calculating a mask vector and an index vector based on the base token and edit token.

Claims

1. 1. A computer-implemented method for image editing, the method comprising: obtaining a base prompt indicating a base image and an edit prompt indicating an edit to be made to the base image; converting the base prompt and edit prompt into a base embedding and an edit embedding, respectively; Over multiple iterations, determining a new edit embedding based on the base embedding and the edit embedding, a time step associated with the iteration, and weights that depend on the time step, the weights controlling the blending of the base embedding and the edit embedding; inputting the base embedding into a diffusion model in a base backward process configured to update a base latent image associated with the base image; and inputting the new edit embeddings into the diffusion model in an edit backward process configured to update an edited latent image associated with an edited image, wherein a cross-attention map generated from the diffusion model in the base backward process is input into the diffusion model in the edit backward process. Repeating and; converting the edited latent image to the edited image; and outputting the edited image. A computer-implemented method.

2. 2. The computer-implemented method of claim 1, wherein the weights depend on the time step such that at earlier time steps the base embedding and the edit embedding blend less than at later time steps.

3. The computer-implemented method of claim 1 , wherein the weights further depend on a time-invariant parameter that controls how much the base embedding and the edit embedding blend.

4. 2. The computer-implemented method of claim 1, wherein converting the base prompts and edit prompts into base embeddings and edit embeddings involves converting the base prompts and edit prompts into base tokens and edit tokens, and converting the base tokens and edit tokens into base embeddings and edit embeddings.

5. The computer-implemented method of claim 4 , wherein the new edit embedding is further based on a mask vector and an index vector calculated based on the base token and the edit token.

6. 2. The computer-implemented method of claim 1, wherein inputting the base embedding into the diffusion model in the base backward process and inputting the new edit embedding into the diffusion model in the edit backward process overlap in time.

7. The computer-implemented method of claim 1 , wherein the base prompt is derived from an image.

8. The computer-implemented method of claim 1 , wherein the base prompt and / or the edit prompt are obtained from user input.

9. A computer program product which, when executed on a computer, causes the computer to carry out a method, the method comprising: obtaining a base prompt indicating a base image and an edit prompt indicating an edit to be made to the base image; converting the base prompt and edit prompt into a base embedding and an edit embedding, respectively; Over multiple iterations, determining a new edit embedding based on the base embedding and the edit embedding, a time step associated with the iteration, and weights that depend on the time step, the weights controlling the blending of the base embedding and the edit embedding; inputting the base embedding into a diffusion model in a base backward process configured to update a base latent image associated with the base image; and inputting the new edit embeddings into the diffusion model in an edit backward process configured to update an edited latent image associated with an edited image, wherein a cross-attention map generated from the diffusion model in the base backward process is input into the diffusion model in the edit backward process. Repeating and; converting the edited latent image to the edited image; and outputting the edited image. Computer program.

10. 1. An information processing device comprising a memory and a processor coupled to the memory, the processor configured to execute a method, the method comprising: obtaining a base prompt indicating a base image and an edit prompt indicating an edit to be made to the base image; converting the base prompt and edit prompt into a base embedding and an edit embedding, respectively; Over multiple iterations, determining a new edit embedding based on the base embedding and the edit embedding, a time step associated with the iteration, and weights that depend on the time step, the weights controlling the blending of the base embedding and the edit embedding; inputting the base embedding into a diffusion model in a base backward process configured to update a base latent image associated with the base image; and inputting the new edit embeddings into the diffusion model in an edit backward process configured to update an edited latent image associated with an edited image, wherein a cross-attention map generated from the diffusion model in the base backward process is input into the diffusion model in the edit backward process. Repeating and; converting the edited latent image to the edited image; and outputting the edited image. Information processing device.

Images