Editing image using image generation model

Abstract

Embodiments of the present disclosure relate to editing an image using an image generation model. A method, apparatus, non-transitory computer readable medium and system for image processing includes obtaining an input image and a modification cue, where the input image depicts an object having a first attribute and the modification cue describes a modification from the first attribute to a second attribute different from the first attribute; encoding the modification cue word to obtain text embedding, wherein the text embedding represents modification in the embedding space; and generating a modified image based on the input image and the text embedding, wherein the modified image depicts the object having the second attribute.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 707,617, filed October 15, 2024, with the United States Patent and Trademark Office, pursuant to 35 U.S. SC § 119, the disclosures of which are incorporated herein by reference in their entirety. Background Technology

[0002] The following generally pertains to image processing, and more specifically to image editing using machine learning models. Image processing refers to using computers to edit images using algorithms or processing networks. In some cases, image processing software can be used for a wide variety of image processing tasks, such as image restoration, image detection, image generation, image compositing, and image editing.

[0003] In some cases, image editing involves using machine learning models to generate modified images based on adjustments to an input image. For example, a machine learning model may be trained to generate an edited image based on text prompts, masked input, and / or the input image itself. In some cases, the edited image may depict modifications to the input image, such as changes from a first element to a second element. Summary of the Invention

[0004] Embodiments of this disclosure provide a method and system for image editing using a generative model. In one aspect, the system generates a modified image based on an input image and a modification cue, the modified image depicting the changes described by the modification cue. In another aspect, the system includes a text encoder configured to encode the modification cue to obtain a text embedding. The system includes an image generation model trained to generate the modified image based on the input image and the text embedding. In embodiments, the image generation model includes a diffusion model configured to guide the image generation process using a cross-attention mechanism. The image generation model generates the modified image by editing the object or attribute described by the modification cue while preserving other visual features of the input image. In some aspects, the image generation model is trained using a first synthetic image generated based on the cue describing the object and a second synthetic image generated based on changes to the cue.

[0005] A method, apparatus, non-transitory computer-readable medium, and system for image processing includes: obtaining an input image and a modification prompt, wherein the input image depicts an object having a first attribute, and the modification prompt describes a modification from the first attribute to a second attribute different from the first attribute; encoding the modification prompt using a text encoder to obtain a text embedding, wherein the text embedding represents the modification in an embedding space; and generating a modified image using an image generation model based on the input image and the text embedding, wherein the modified image depicts an object having the second attribute.

[0006] A method, apparatus, non-transitory computer-readable medium, and system for image processing include: obtaining an input image and a modification prompt, the modification prompt indicating a modification to the input image; and generating a modified image using an image generation model based on the input image and the modification prompt, wherein the modified image depicts content from the input image with modifications from the modification prompt, and wherein the image generation model is trained using a first predicted image generated based on captions and a second predicted image generated based on changes to the captions.

[0007] A method, apparatus, non-transitory computer-readable medium, and system for training a machine learning model include: obtaining a training set comprising training images, modification cue words describing changes to the training images, and modified training images depicting the changes to the training images; encoding the modification cue words using a text encoder to obtain text embeddings, wherein the text embeddings represent modifications in an embedding space; and training an image generation model using the training set and the text embeddings to generate synthetic images based on the modification cue words.

[0008] An apparatus and system for image processing includes a memory component and a processing device coupled to the memory component, the processing device being configured to perform operations including: acquiring an input image and a modification prompt, wherein the input image depicts an object having a first attribute, and the modification prompt describes a modification from the first attribute to a second attribute different from the first attribute; encoding the modification prompt using a text encoder to obtain a text embedding, wherein the text embedding represents the modification in an embedding space; and generating a modified image based on the input image and the text embedding using an image generation model, wherein the modified image depicts an object having the second attribute. Attached Figure Description

[0009] Figure 1 Examples of image processing systems according to various aspects of this disclosure are shown.

[0010] Figure 2 Examples of methods for conditional image editing according to various aspects of this disclosure are shown.

[0011] Figure 3 Examples of methods for generating modified images according to various aspects of this disclosure are shown.

[0012] Figure 4 Examples of image editing using text prompts according to various aspects of this disclosure are shown.

[0013] Figure 5 An example of output image refinement according to various aspects of this disclosure is shown.

[0014] Figure 6 Examples of machine learning models based on various aspects of this disclosure are shown.

[0015] Figure 7 Examples of image generation systems according to various aspects of this disclosure are shown.

[0016] Figure 8 Examples of image generation models based on various aspects of this disclosure are shown.

[0017] Figure 9 An example of the U-Net architecture according to various aspects of this disclosure is shown.

[0018] Figure 10 An example of a diffusion converter model according to various aspects of this disclosure is shown.

[0019] Figure 11 Examples of diffusion processes according to various aspects of this disclosure are shown.

[0020] Figure 12 Examples of methods for training machine learning models according to various aspects of this disclosure are shown.

[0021] Figure 13 Examples generated from training data based on various aspects of this disclosure are shown.

[0022] Figure 14 An example flowchart of a step-by-step procedure is shown in an example implementation of an algorithm that can perform operations for training a machine learning model, according to various aspects of this disclosure.

[0023] Figure 15 Examples of methods for training diffusion models according to various aspects of this disclosure are shown.

[0024] Figure 16 Examples of image processing apparatuses according to various aspects of this disclosure are shown.

[0025] Figure 17 Examples of computing devices according to various aspects of this disclosure are shown. Detailed Implementation

[0026] Embodiments of this disclosure relate to image editing using generative machine learning. Some embodiments of this disclosure relate to an image generation system that accurately generates a modified image depicting modifications from a first attribute to a second attribute of an object in an input image. In some cases, the modified image depicts modifications from a first object to a second object while preserving the attributes of the first object. In some aspects, the system includes a text encoder configured to generate text embeddings based on modification cues. The text embeddings are provided to the system's image generation model to ensure that the synthesized image accurately depicts the modifications to the attribute or object described by the modification cues.

[0027] In the field of image editing, particularly in the modification of image elements, machine learning systems are used to replace one image element (e.g., an attribute or object) with another, using text prompts. Traditional image editing systems employ multiple inputs to tune the model to generate the edited image. These inputs may include the input image, text prompts, and a mask input. However, in some cases, these systems can alter image elements not described by text prompts. Therefore, these systems cannot preserve the image characteristics of the image elements.

[0028] Some traditional image generation systems are configured to edit images based on specific instructions. However, in some cases, these systems may overinterpret ambiguous instructions or misinterpret complex instructions, resulting in a misalignment between the edit and the user's intent. In some cases, these systems generate images with a loss of image quality. For example, these systems use iterative editing processes, which can reduce the overall quality of the output image and introduce artifacts or blurred details.

[0029] Some systems are configured to edit the attributes or visual features of objects depicted in an input image. For example, visual features of an object may include color, texture, shape, contrast, brightness, pattern, edges, and / or orientation. However, when editing an image, these systems may not be able to preserve the texture or lighting of the object, especially in complex areas of the image. In some cases, the edited object or feature may look unnatural or be poorly integrated into the scene of the image, reducing the overall aesthetic appeal of the output image.

[0030] Accordingly, this disclosure provides systems and methods for improving conventional image generation systems by precisely editing the image elements described by the modification prompt while preserving the attributes of the image elements. For example, when given the prompt stating "Change the spoon to a fork" and an input image depicting a spoon, the modified image depicts a fork with the same hue and texture as the spoon depicted in the input image. This is achieved using a system that includes an image generation model that uses the input image to initiate the image generation process and uses the text embedding of the modification prompt as guidance.

[0031] According to some aspects, the system receives an input image and modification prompts to generate an edited image (e.g., a modified image). In one aspect, the system includes a text encoder configured to generate text embeddings based on the modification prompts. In some aspects, the text embedding represents a modification from one image element (e.g., an attribute or object) to another image element. By using text embeddings to guide the image generation process, the image generation model is able to accurately identify the image elements within the input image to be modified.

[0032] According to some aspects, the system includes an image generation model trained to generate edited images (e.g., modified images) based on an input image and text prompts. By initializing the image generation process using the input image, the image generation model is able to accurately preserve other image elements while editing the image elements described by the text prompts.

[0033] refer to Figure 1 and Figure 17 An example system of inventive concepts in image processing is provided. (Reference) Figure 2 and Figures 4 to 5 This provides example applications of inventive concepts in image processing. (Reference) Figures 6 to 10 and Figure 16 Details regarding the architecture of the image processing device are provided. (Reference) Figure 3 and Figure 11 Examples of image processing procedures are provided. (Reference) Figures 12 to 15 A description of an example training process is provided.

[0034] Accordingly, embodiments of this disclosure generate edited images more accurately. In some aspects, the system of this disclosure can be used with various types of diffusion models, including pixel-based diffusion models or latent-based diffusion models. According to some aspects, the system is jointly trained with multiple tasks including inpainting, expansion, editing tasks, segmentation, depth estimation, normal vector estimation, colorization, and low-level vision. In some cases, editing tasks include recoloring, retexturing, structural editing, appearance editing, text editing, global editing, and style editing. In some aspects, the image generation model can be further guided by reference inputs such as fine-grained color images, texture images, and reference images. In some aspects, the image generation model can be pre-trained, thereby reducing training costs. In some embodiments, the image generation model uses dual classifier-free guidance (CFG) to enhance the image quality of the modified image.

[0035] As used herein, the input image, image, reference image, or training image depicts one or more objects, elements, or scenes. The image serves as the visual basis for generating the modified image. The input image may include visual information such as the shape, color, texture, and spatial arrangement of elements.

[0036] Modification prompts are text-based instructions that describe the target changes to be applied to an input image. Modification prompts can describe a transformation from a first attribute or object to a second attribute or object. For example, a modification prompt might state "change the hat to a helmet" or "change the blue sky to sunset orange."

[0037] An object refers to a visible visual component depicted in an image, which can include tangible items (such as a person, chair, or tree) or conceptual elements (such as a scene or background). During the image editing process, objects can be the target of modification or preservation.

[0038] An attribute, primary attribute, secondary attribute, or tertiary attribute refers to a characteristic, feature, or visual characteristic of an object depicted in an input image. Attributes can include tangible features (such as shape, size, color, and structure) as well as intangible elements (such as lighting, texture, shadows, gloss, and contrast). In some cases, an attribute can refer to an object, element, or scene in the image (e.g., an object depicted in the input image).

[0039] An embedding is a numerical vector representation of input data in a continuous low-dimensional space used to perform machine learning tasks. Text embeddings are vector representations of modified prompts or other text inputs, capturing the semantic meaning of the modified prompts. Text embeddings are generated using a text encoder and used to guide image generation models. Image embeddings are numerical representations of visual features extracted from images (e.g., input images, reference images). Image embeddings capture elements such as shape, color, texture, and structure and are used to modulate or guide the image generation process.

[0040] The latent space refers to the continuous, multi-dimensional space in which embeddings are represented. The latent space encodes semantic or visual features in a compact form, allowing for efficient manipulation and processing. In image generation, both text and image embeddings reside in the latent space (or multimodal space) to guide output generation.

[0041] Masked input refers to a binary or multi-valued image of the input image that indicates a specific region of the image to be edited. A masking system applies changes within the masked region while preserving other areas of the image.

[0042] Reference input is an auxiliary input image used to guide the style, texture, color, depth, or structure of the generated image. Examples of reference input include style images, color images, depth maps, texture maps, etc. A style image can be an image depicting an artistic style. A color image can be an image depicting a target color or palette. A depth map can be a grayscale image representing depth information for spatial arrangement. A texture map can be an image including surface-level patterns or textures to be transformed.

[0043] Image features refer to abstract representations of visual characteristics extracted from an image using convolutional or transformer-based encoders. These features include spatial, textural, and semantic information, and are used by image generation models to produce modified images. In some cases, image features can be represented in a latent space or embedding space. Image editing

[0044] exist Figures 1 to 5 and Figure 11 Among them, a method, apparatus, non-transitory computer-readable medium, and system for image processing include: obtaining an input image and a modification prompt, wherein the input image depicts an object having a first attribute, and the modification prompt describes a modification from the first attribute to a second attribute different from the first attribute; encoding the modification prompt using a text encoder to obtain a text embedding, wherein the text embedding represents the modification in an embedding space; and generating a modified image using an image generation model based on the input image and the text embedding, wherein the modified image depicts an object having the second attribute.

[0045] In some aspects, the modified image retains the third property of the object. Examples of methods, apparatuses, non-transitory computer-readable media, and systems also include obtaining a mask input indicating a region of an input image, wherein the modified image is generated based on the mask input. Examples of methods, apparatuses, non-transitory computer-readable media, and systems also include obtaining a reference input. Some examples further include generating a reference embedding based on the reference input, wherein the modified image is generated based on the reference embedding.

[0046] Examples of methods, apparatuses, non-transitory computer-readable media, and systems include obtaining a noisy input. Some examples also include denoising the noisy input based on text embedding to obtain a modified image. Examples of methods, apparatuses, non-transitory computer-readable media, and systems also include modifying the color of an input image to obtain a modified input image, wherein the modified image is generated based on the modified input image.

[0047] Examples of methods, apparatuses, non-transitory computer-readable media, and systems include generating a first image feature based on an input image. Some examples also include generating a second image feature based on a text embedding. Some examples further include combining the first and second image features to obtain combined image features, wherein a modified image is generated based on the combined image features.

[0048] In some aspects, a training set is used to train an image generation model to edit images. This training set includes training images depicting objects with a first attribute and training cue words describing modifications from the first attribute to a second attribute. In other aspects, the image generation model is trained using a first predicted image generated based on captions and a second predicted image generated based on changes to the captions.

[0049] A method, apparatus, non-transitory computer-readable medium, and system for image processing include: acquiring an input image and a modification prompt, the modification prompt indicating a modification to the input image; and generating a modified image using an image generation model based on the input image and the modification prompt, wherein the modified image depicts content from the input image with modifications from the modification prompt, and wherein the image generation model is trained using a first predicted image generated based on captions and a second predicted image generated based on changes to the captions.

[0050] In some respects, the modified image retains the third attributes of the object. Some examples of methods, apparatuses, non-transitory computer-readable media, and systems also include obtaining a mask input indicating a region of an input image, based on which the modified image is generated.

[0051] Examples of methods, apparatuses, non-transitory computer-readable media, and systems also include obtaining reference input. Some examples also include generating a reference embedding based on the reference input, wherein a modified image is generated based on the reference embedding.

[0052] Examples of methods, apparatuses, non-transitory computer-readable media, and systems include obtaining a noisy input. Some examples also include denoising the noisy input based on text embedding to obtain a modified image. Examples of methods, apparatuses, non-transitory computer-readable media, and systems also include modifying the color of an input image to obtain a modified input image, wherein the modified image is generated based on the modified input image.

[0053] Examples of methods, apparatuses, non-transitory computer-readable media, and systems include generating a first image feature based on an input image. Some examples also include generating a second image feature based on a text embedding. Some examples further include combining the first and second image features to obtain combined image features, wherein a modified image is generated based on the combined image features.

[0054] Figure 1 Examples of image processing systems according to various aspects of this disclosure are shown. The illustrated examples include a user 100, a user device 105, an image processing apparatus 110, a cloud 115, a database 120, and a display device 125. In some aspects, the user device 105 includes the display device 125. The image processing apparatus 110 is a reference. Figure 16 Examples of the corresponding elements described, or including references Figure 16 The various aspects of the corresponding element described.

[0055] refer to Figure 1 User 100 provides input text prompts (e.g., modification prompts) and input images to image processing device 110 via user device 105 and cloud 115. In some cases, the text prompts describe a modification from a first attribute to a second attribute or from a first object to a second object. For example, the text prompt states "replace the basket with a light blue plate." For example, the input image depicts several breakfast breads in a white plastic basket. In some cases, image processing device 110 includes a machine learning model that generates a modified image depicting the change from one object to another (e.g., from a basket to a light blue plate) based on the text prompts and the input image. In some cases, the object's attributes (e.g., plastic appearance or texture) are preserved in the modified image.

[0056] In some aspects, the image processing apparatus 110 includes a text encoder configured to generate text embeddings based on text prompts. In some cases, the text embeddings represent modifications from a basket to a light blue plate. In some aspects, the image processing apparatus 110 includes an image generation model trained to generate a modified image depicting the modifications. For example, an input image is combined with input noise to initiate the image generation process. The text embeddings are then used to guide the image generation process, wherein the image generation model generates an output representing changes to the elements described by the text prompts (e.g., output image features). In some aspects, an image decoder decodes the output to generate the modified image. The image processing apparatus 110 displays the modified image to a user 100 via a display device 125 of a user device 105 via a cloud 115.

[0057] User device 105 may be a personal computer, laptop computer, mainframe computer, handheld computer, personal assistant, mobile device, or any other suitable processing device. In some examples, user device 105 includes software containing an image processing application. In some examples, the image processing application on user device 105 may include the functionality of image processing device 110.

[0058] The user interface enables user 100 to interact with user device 105. In some embodiments, the user interface may include an audio device (such as an external speaker system), an external display device (such as a display screen), or an input device (e.g., a remote control device that interfaces directly with the user interface or via an I / O controller module). In some cases, the user interface may be a graphical user interface (GUI). In some examples, the user interface may be represented by code, which is sent to user device 105 and rendered natively by a browser. References Figure 2 The process of using the image processing device 110 is further described.

[0059] According to some aspects, the image processing apparatus 110 includes a computer-implemented network, which includes a machine learning model, a text encoder, an image encoder, and an image generation model. The image processing apparatus 110 also includes a processor unit, a memory unit, an I / O module, and a training component. In some embodiments, the image processing apparatus 110 further includes, as referenced... Figure 16 The described communication interface, user interface components, and bus. Additionally or alternatively, the image processing apparatus 110 communicates with the user equipment 105 and the database 120 via a cloud 115. The image processing apparatus 110 is a reference. Figure 6 Examples of the corresponding elements described, or including references Figure 6 The description covers various aspects of the corresponding element. (See reference) Figure 2 Further details regarding the operation of the image processing apparatus 110 are described.

[0060] In some cases, the image processing apparatus 110 is implemented on a server. The server provides one or more functions to users linked through one or more networks of various kinds. In some cases, the server includes a single microprocessor board comprising a microprocessor responsible for controlling various aspects of the server. In some cases, the server uses the microprocessor and protocols to exchange data with other devices / users on one or more networks via Hypertext Transfer Protocol (HTTP) and Simple Mail Transfer Protocol (SMTP), although other protocols such as File Transfer Protocol (FTP) and Simple Network Management Protocol (SNMP) may also be used. In some cases, the server is configured to send and receive files in Hypertext Markup Language (HTML) format (e.g., for displaying web pages). In various embodiments, the server includes a general-purpose computing device, a personal computer, a laptop computer, a mainframe computer, a supercomputer, or any other suitable processing device.

[0061] Cloud 115 is a computer network configured to provide on-demand availability of computer system resources, such as data storage and computing power. In some examples, Cloud 115 provides resources that do not require active management by users (e.g., user 100). The term cloud is sometimes used to describe data centers on the Internet that are available to many users. Some large cloud networks have capabilities distributed across multiple locations originating from a central server. Servers are designated edge servers if they have direct or close connections to users. In some cases, Cloud 115 is limited to a single organization. In other examples, Cloud 115 can be used by many organizations. In one example, Cloud 115 includes a multi-tiered communication network comprising multiple edge routers and a core router. In another example, Cloud 115 is based on a local collection of switches in a single physical location.

[0062] Depending on some aspects, database 120 stores training data (or training sets), which includes training images depicting objects with a first attribute and training cue words describing modifications from the first attribute to a second attribute different from the first attribute. Database 120 is an organized collection of data. For example, database 120 stores data in a specified format referred to as a schema. Database 120 can be constructed as a single database, a distributed database, multiple distributed databases, or an emergency backup database. In some cases, a database controller can manage the data storage and processing in database 120. In some cases, a user (e.g., user 100) interacts with the database controller. In other cases, the database controller can operate automatically without user interaction.

[0063] Figure 2 Examples of a method 200 for conditional image editing according to aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor with functional elements that execute a set of code to control means. Additionally or alternatively, dedicated hardware is used to perform certain processes. Generally, these operations are performed according to the methods and processes described according to aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0064] At step 205, the system provides text prompts and an input image. In some cases, this step can be referenced or can be found in a reference book. Figure 1 The user is described in the text. In some cases, the user provides text prompts describing modifications from one image element to another. In other cases, for example, the input image depicts image elements.

[0065] At operation 210, the system generates a text conditional guidance embedding. In some cases, this step involves referencing... Figure 1 and16 The described image processing apparatus may be used to perform this step. In some cases, the operation of this step involves reference to... Figures 6 to 8 and Figure 16 The described text encoder may be executed by it. In some cases, the system includes a text encoder configured to encode text prompts to generate text embeddings. For example, text embeddings are used as guides to direct the image generation process of an image generation model. In some cases, text embeddings are combined with features from a U-Net within the image generation model via a cross-attention layer. (Reference) Figure 8 and Figure 10 Further details about U-Net are described.

[0066] At operation 215, the system initializes the noise input. In some cases, this step involves referencing... Figure 1 and Figure 16 The described image processing apparatus may be used to perform this step. In some cases, the operation of this step involves reference to... Figure 6 and Figure 16 The described image generation model can be executed by it. In some cases, initialization includes a noisy input containing random noise. The noisy input can be in the latent space. By initializing the image generation model with random noise, it is possible to generate different variations of the synthetic image. In some cases, cross-attention blocks within the image generation model can be used to combine conditional embeddings (such as text encoding or text embedding) with noisy features to guide the image generation process.

[0067] In some embodiments, a noisy input is combined with an input image to obtain a noisy image, wherein the noisy image is used to initiate the image generation process. For example, the noisy image includes visual features of one or more image elements depicted in the input image. By initializing the image generation model using a noisy image, one or more image elements can be preserved in the output image. (Reference) Figure 8 Further details about the image generation process are described.

[0068] At step 220, the system generates media content. In some cases, this step involves referencing... Figure 1 and Figure 16 The described image processing apparatus may be used to perform this step. In some cases, the operation of this step involves reference to... Figure 6 and Figure 16 The described image generation model can be executed by it. In some cases, media content includes modified images. For example, a modified image depicts changes to image elements described by text prompts.

[0069] Figure 3Examples of a method 300 for generating a modified image according to aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor with functional elements that execute a set of code to control means. Additionally or alternatively, dedicated hardware is used to perform certain processes. Generally, these operations are performed according to the methods and processes described according to aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0070] At operation 305, the system obtains an input image and a modification prompt, wherein the input image depicts an object having a first attribute, and the modification prompt describes a modification from the first attribute to a second attribute different from the first attribute. In some cases, this step involves referencing... Figure 6 and Figure 16 The described image generation model or the model that can be executed by it. In some cases, modification prompts are text prompts describing a change from one image element to another. In some cases, image elements include attributes or objects. For example, attributes can describe the visual appearance or image features that constitute the overall composition of the image (such as subject, shape, color, texture, pattern, background scene, visual attributes, and / or style). In some cases, objects include people, animals, and inanimate objects (such as tables, chairs, plants, etc.). In some embodiments, modification prompts can describe a modification from a first object to a second object different from the first object.

[0071] At operation 310, the system encodes the modification prompt to obtain a text embedding, where the text embedding represents the modification in the embedding space. In some cases, this step involves referencing... Figures 6 to 8 and Figure 16 The described text encoder or can be executed by it. In some embodiments, the system generates image embeddings based on reference inputs or reference images. In some cases, text embeddings and image embeddings are used to guide the image generation process.

[0072] In some cases, text embeddings are numerical vectors that capture the semantics of text, encoding words, phrases, or sentences into a dense, continuous space. For example, text embeddings are encoded into a text embedding space, which is a low-dimensional vector space. Text embeddings are generated by passing text cue words through an encoder (e.g., a text encoder or multimodal encoder) that learns relationships between words based on context within a large corpus of text. In some cases, text embeddings represent textual features of text cue words (e.g., semantic meaning, relationships between words, or lexical features).

[0073] In some cases, the text embedding space is a continuous low-dimensional vector space, where each vector represents the semantic meaning of the text. Organizing the points in the text embedding space such that texts with similar meanings are located close to each other allows for context-based use to reflect relationships between different words, phrases, or sentences.

[0074] For example, image embeddings capture the fundamental visual features or characteristics of an image (such as color, texture, shape, and spatial relationships). In some aspects, a transformer prior model is trained to generate image embeddings based on text cue words, where the image embeddings include visual features of the image elements described by the text cue words.

[0075] In some cases, the image embedding space is a high-dimensional vector space where each point corresponds to a visual representation of the image. In this space, the distance between points reflects the similarity of the images' visual features. In some cases, similar images are positioned closer together based on the features encoded in the image embeddings. In other cases, text embeddings and image embeddings are combined in a joint embedding space.

[0076] At operation 315, the system generates a modified image based on the input image and text embedding, wherein the modified image depicts an object with a second attribute. In some cases, this step involves referencing... Figure 6 and Figure 16 The described image generation model can be executed by it. In some cases, the modified image includes image pixels from the input image and image pixels generated by the image generation model.

[0077] Figure 4 Examples of image editing using text prompts according to various aspects of this disclosure are shown. The examples shown include an image editing system 400, an input image 405, a modification prompt 410, a mask 415, a machine learning model 420, a modified image 425, and a conventional modified image 430. In some embodiments, the image editing system 400 is implemented in a user interface.

[0078] refer to Figure 4 The image editing system 400 receives an input image 405 and a modification prompt 410, and generates a modified image 425. In some embodiments, the image editing system 400 receives the input image 405, the modification prompt 410, and a mask 415 as input to generate the modified image 425. In some embodiments, the image editing system 400 receives one or more reference inputs to generate the modified image 425. For example, the reference inputs may include a fine-grained color image, a texture image, and a reference image.

[0079] According to some embodiments, machine learning model 420 receives an input image 405 and a modification prompt 410 as input. For example, the input image 405 depicts two cooked steaks placed on a grill. For example, the modification prompt 410 describes a modification such as "change the meat to raw" (e.g., a change in image elements). In some aspects, machine learning model 420 includes a text encoder configured to encode the modification prompt 410 to generate a text embedding. In some aspects, machine learning model 420 includes an image generation model trained to generate the modified image 425 based on the input image 405 and the text embedding. For example, the image generation process is initialized using the input image 405 and guided by the text embedding. When the image generation process is initialized using the input image 405, image elements described by the modification prompt 410 (e.g., cooked steaks) are changed while other image elements of the input image 405 (such as color, shape, texture, etc.) are preserved. For example, the shape of the cooked steaks is preserved.

[0080] According to some embodiments, a modified image 425 is further generated based on mask 415. For example, mask 415 indicates regions (e.g., coarse or fine regions) of the input image 405 that depict image elements (e.g., cooked steak). By using mask 415 to guide the image generation process, the accuracy of the image generation model can be further improved.

[0081] A conventional image generation system receives a text prompt describing modifications to image elements and an input image 405 depicting those elements to generate a conventional modified image 430. In some cases, the input image 405 is combined with a mask (e.g., mask 415) to generate a masked image, which is used to initialize the image generation process. However, a conventional system may not be able to accurately generate the correct pixels in the area indicated by mask 415. Accordingly, new pixels may be generated, thereby altering the image's attributes. For example, the shape and size of the steak depicted in the conventional modified image 430 differ from those depicted in the input image 405, while the shape and size of the steak depicted in the modified image 425 are preserved.

[0082] Image editing system 400 is a reference Figure 5 Examples of the corresponding elements described, or including references Figure 5 The description covers various aspects of the corresponding element. Input image 405 is a reference. Figures 5 to 7 Examples of the corresponding elements described, or including references Figures 5 to 7 The description covers all aspects of the corresponding element. Modification prompt 410 is for reference. Figure 5 and Figure 6Examples of the corresponding elements described, or including references Figure 5 and Figure 6 The various aspects of the corresponding element described.

[0083] Mask 415 is a reference Figure 13 Examples of the corresponding elements described, or including references Figure 13 The description covers various aspects of the corresponding element. Machine learning model 420 is a reference. Figure 5 Examples of the corresponding elements described, or including references Figure 5 The description covers various aspects of the corresponding element. Modified image 425 is for reference. Figure 5 and Figure 6 Examples of the corresponding elements described, or including references Figure 5 and Figure 6 The various aspects of the corresponding element described.

[0084] Figure 5 An example of output image refinement according to various aspects of this disclosure is shown. The example shown includes an image editing system 500, an input image 505, a modification prompt 510, a machine learning model 515, a modified image 520, and a refined image 525. In some embodiments, the image editing system 500 is implemented in a user interface.

[0085] refer to Figure 5 Image editing system 500 receives input image 505 and modification prompts 510 and generates a refined image 525. In some embodiments, machine learning model 515 receives input image 505 and modification prompts 510 and generates a modified image 520. The process of generating the modified image 520 is essentially the same as... Figure 4 The process described herein is the same. According to some embodiments, the modified image 520 is input into a machine learning model 515 to generate a refined image 525. For example, the machine learning model 515 generates input image features based on the input image 505, and generates modified image features based on the modified image 520. By interpolating image features, the image generation model of the machine learning model 515 is able to generate a refined image 525 with enhanced identity preservation. For example, the shapes of the stool sofa and the blue-green sofa surrounding the blue-green sofa remain unchanged, while the color of the blue-green sofa is modified based on the modification cue word 510.

[0086] Image editing system 500 is a reference Figure 4 Examples of the corresponding elements described, or including references Figure 4 The description covers various aspects of the corresponding element. Input image 505 is a reference. Figure 4 , Figure 6 and Figure 7Examples of the corresponding elements described, or including references Figure 4 , Figure 6 and Figure 7 The description covers all aspects of the corresponding element. Modification prompt 510 is for reference. Figure 4 and Figure 6 Examples of the corresponding elements described, or including references Figure 4 and Figure 6 The various aspects of the corresponding element described.

[0087] Machine learning model 515 is a reference Figure 4 Examples of the corresponding elements described, or including references Figure 4 The description covers various aspects of the corresponding element. The modified image 520 is for reference. Figure 4 and Figure 6 Examples of the corresponding elements described, or including references Figure 4 and Figure 6 The various aspects of the corresponding element described. System Architecture

[0088] exist Figures 6 to 10 and Figures 16 to 17 In the present invention, an apparatus and system for image processing includes a memory component and a processing device coupled to the memory component, the processing device being configured to perform operations including: acquiring an input image and a modification prompt, wherein the input image depicts an object having a first attribute, and the modification prompt describes a modification from the first attribute to a second attribute different from the first attribute; encoding the modification prompt using a text encoder to obtain a text embedding, wherein the text embedding represents the modification in an embedding space; and generating a modified image using an image generation model based on the input image and the text embedding, wherein the modified image depicts an object having the second attribute.

[0089] In some aspects, the image generation model includes a diffusion model. In some aspects, the modified image retains the third property of the object. Some examples of the apparatus and system also include an image encoder configured to generate a reference embedding based on a reference input, wherein the modified image is generated based on the reference embedding.

[0090] Figure 6 Examples of machine learning models according to various aspects of this disclosure are shown. The examples shown include a machine learning system 600, a modified prompt word 605, a text encoder 610, a text embedding 615, a reference input 620, an image encoder 625, a reference embedding 630, an input image 635, a masked input 640, an image generation model 645, and a modified image 650. In some aspects, the machine learning system 600 includes a text encoder 610, an image encoder 625, and an image generation model 645.

[0091] refer to Figure 6 According to some embodiments, machine learning system 600 receives a modification prompt 605 and an input image 635 and generates a modified image 650. For example, text encoder 610 receives the modification prompt 605 stating "change meat to raw" and generates a text embedding 615 representing the modification in the embedding space. The text embedding 615 is provided to image generation model 645 to guide the image generation process. In some cases, input image 635 is provided to image generation model 645 to initiate the image generation process.

[0092] For example, the input image 635 is combined with a noisy input to initialize the backdiffusion process of the image generation model 645. In some embodiments, the input image 635 is used as a guide to direct the image generation process. For example, an image embedding is generated based on the input image 635, and the image embedding is combined with a text embedding 615 to guide the image generation process. (See reference) Figure 8 Further details regarding the guided embedding are described. In one aspect, the image generation model 645 generates a modified image 650 based on the text embedding 615 and the input image 635. (Reference) Figure 7 Further details are described regarding the image generation process using image generation model 645.

[0093] In some embodiments, the image generation model 645 receives one or more reference inputs (e.g., reference input 620) to further guide the image generation process. In some cases, for example, reference input 620 includes a fine-grained color image, a texture image, or a style image. For example, image encoder 625 receives reference input 620 and generates a reference embedding 630. In some cases, the reference embedding is an image embedding. In some embodiments, reference embedding 630 is combined with text embedding 615 to guide the image generation process.

[0094] In some embodiments, the machine learning system 600 performs global style transfer to generate a modified image 650. For example, a training-free algorithm is derived by scaling the activation of modality-specific learnable attention and applying inversion to preserve the identification of objects depicted in the input image 635. In some embodiments, an image generation model 645 is trained using modality-specific attention. In some aspects, the image generation model 645 is tuned using textual prompts (e.g., modified prompt 605) and style images (e.g., reference input 620) such that the image generation model 645 can generate styled outputs accordingly as each modality-specific attention output is scaled. For example, as referenced... Figure 9As described, modal attention from the image (e.g., modal attention of reference embedding 630) is scaled down in medium-resolution or lower-resolution layers of the U-Net. Additionally, high-resolution layers of the U-Net are scaled up. In some cases, inversion is applied to generate latent features (e.g., representations in the latent space) of the input image 635, and the system denoises this latent feature back into the image (e.g., modified image 650), while conditioned on a style reference image (e.g., reference input 620).

[0095] In some embodiments, the machine learning system 600 performs fine-grained color control for editing the colors of an input image 635. For example, an image generation model 645 receives color scales with predetermined colors as a reference (e.g., reference input 620). For example, a color scale is a small, defined region of uniform color representing a normalized color. An image encoder 625 (or a multi-mode encoder) encodes the visual features of the color scales to obtain a reference embedding 630. In some cases, the reference embedding 630 of the color scale includes precise color information (e.g., representing hexadecimal codes or RGB values). The image generation model 645 uses the reference embedding 630 as guidance to generate a modified image 650, where the colors (the colors of the entire image or a portion of the image) are controlled by hexadecimal codes or RGB values.

[0096] In some embodiments, the image generation model 645 also receives a mask input 640 to generate a modified image 650. For example, the mask input 640 indicates a region of an object or an attribute of an object to be modified (e.g., a coarse or fine region). In some cases, the image generation model 645 is capable of identifying regions unrelated to the mask input 640. In some cases, a mask embedding is generated based on the mask input 640, and the mask embedding is combined with convolutional layers in the residual blocks of the decoder of the U-Net of the image generation model 645. (See reference...) Figure 8 Further details are described regarding the image generation process using a masked input of 640.

[0097] According to some embodiments, the machine learning system 600 performs dual classifier-free guidance (CFG) to further enhance image quality. CFG is a technique used to enhance the image quality in generated images using a diffusion-based image generation model. For example, in CFG, the model generates an image by balancing two outputs: one output is conditional on a cue word (e.g., a text cue word), while the other output is unconditional. In dual CFG, two guidance signals are used. For example, a first guidance signal is used to guide image generation based on a specific input (e.g., text, image, mask, reference image, color, etc.), while a second guidance signal manipulates the model away from undesirable outputs, thus providing finer control over the results. Dual CFG enhances the fidelity of the generated image by maintaining consistency with the cue word while preventing the generation of unrealistic or unwanted elements.

[0098] Modifying the prompt word 605 is for reference. Figure 4 and Figure 5 Examples of the corresponding elements described, or including references Figure 4 and Figure 5 The description covers various aspects of the corresponding element. The text encoder 610 is a reference. Figure 7 , Figure 8 and Figure 16 Examples of the corresponding elements described, or including references Figure 7 , Figure 8 and Figure 16 The description covers various aspects of the corresponding element. Text embedding 615 is a reference. Figure 7 Examples of the corresponding elements described, or including references Figure 7 The description covers various aspects of the corresponding element. The image encoder 625 is a reference. Figure 8 and Figure 16 Examples of the corresponding elements described, or including references Figure 8 and 16 The various aspects of the corresponding element described.

[0099] Input image 635 is the reference. Figure 4 , Figure 5 and Figure 7 Examples of the corresponding elements described, or including references Figure 4 , Figure 5 and Figure 7 The description covers various aspects of the corresponding element. The mask input 640 is for reference. Figure 7 Examples of the corresponding elements described, or including references Figure 7 The description covers various aspects of the corresponding elements. Image generation model 645 is a reference. Figure 16 Examples of the corresponding elements described, or including references Figure 16 The described aspects of the corresponding element. The modified image 650 is for reference. Figure 4 and Figure 5 Examples of the corresponding elements described, or including references Figure 4 and Figure 5 The various aspects of the corresponding element described.

[0100] Figure 7 Examples of an image generation system 700 according to various aspects of this disclosure are shown. The illustrated examples include an image generation system 700, a mask 705, an input image 710, a noisy input 715, a U-Net 720, text prompts 725, a text encoder 730, a text embedding 735, an attention block 740, a time step 755, a time step embedding 760, a mask input 765, a mask embedding 770, a combined embedding 775, a residual block 780, and output features 785. In one aspect, the attention block 740 includes a self-attention layer 745 and a cross-attention layer 750.

[0101] In some respects, Figure 7 An image generation process with a diffusion time step is illustrated. In some aspects, the image generation process is performed iteratively, where the output features are used as input for subsequent image generation iterations. (Reference) Figure 7 The image generation system 700 receives an input image 710 and a text prompt 725 and generates output features 785. For example, the input image 710 is combined with a noise input 715 to obtain a noisy image, which is provided to the U-Net 720 of the image generation model to initialize the image generation process. In some embodiments, a mask 705 is also provided to the U-Net 720.

[0102] In some embodiments, text prompt word 725 (e.g., reference) Figures 4 to 6 The described modified cue words are provided to the text encoder 730 to generate a text embedding 735. The text embedding 735 is combined with features generated by U-Net 720 via cross-attention. For example, U-Net 720 includes one or more attention blocks, where each attention block 740 includes a self-attention layer 745 and a cross-attention layer 750. The text embedding 735 is added to intermediate features generated based on the input image 710 and the noisy input 715 via the cross-attention mechanism in the cross-attention layer 750.

[0103] In some embodiments, a time step 755 is combined with intermediate output features of U-Net 720. For example, a time step embedding 760 is obtained based on time step 755, wherein the time step embedding 760 is provided to the residual block 780 of U-Net 720. In some embodiments, a mask input 765 is combined with intermediate output features of U-Net 720. For example, a mask embedding 770 is obtained based on mask input 765, wherein the mask embedding 770 is combined with the time step embedding 760 to generate a combined embedding 775. In some cases, the combined embedding 775 is provided to the residual block 780 of U-Net 720. In some cases, intermediate output features are upsampled to generate output features 785. According to some embodiments, output features 785 are used as input to U-Net 720 for subsequent image generation steps.

[0104] Mask 705 is a reference Figure 6 Examples of the corresponding elements described, or including references Figure 6 The description covers various aspects of the corresponding element. Input image 710 is a reference. Figures 4 to 6 Examples of the corresponding elements described, or including references Figures 4 to 6 The description covers various aspects of the corresponding element. U-Net 720 is a reference. Figure 9 Examples of the corresponding elements described, or including references Figure 9 The various aspects of the corresponding element described.

[0105] Text prompt word 725 is for reference. Figure 8 Examples of the corresponding elements described, or including references Figure 8 The description covers various aspects of the corresponding element. Text encoder 730 is a reference. Figure 6 , Figure 8 and Figure 16 Examples of the corresponding elements described, or including references Figure 6 , Figure 8 and Figure 16 The description covers various aspects of the corresponding element. Text embedding 735 is a reference. Figure 6 Examples of the corresponding elements described, or including references Figure 6 The various aspects of the corresponding element described.

[0106] Mask 705 is a reference Figure 6 Examples of the corresponding elements described, or including references Figure 6 The description covers various aspects of the corresponding element. Output feature 785 is a reference. Figure 9 Examples of the corresponding elements described, or including references Figure 9 The various aspects of the corresponding element described.

[0107] Figure 8Examples of image generation models according to various aspects of this disclosure are shown. The examples shown include a diffusion model 800, an original image 805, a pixel space 810, an image encoder 815, original image features 820, a latent space 825, a forward diffusion process 830, noise features 835, a backward diffusion process 840, denoised image features 845, an image decoder 850, an output image 855, text prompts 860, a text encoder 865, guiding features 870, and a guiding space 875.

[0108] Diffusion models are a class of generative neural networks that can be trained to generate new data with features similar to those found in the training data. Specifically, diffusion models can be used to generate new images. Diffusion models can be used for a wide range of image generation tasks, including image super-resolution, image generation with perceptual metrics, conditional generation (e.g., text-guided, color-guided, style-guided, and image-guided generation), image inpainting, and image manipulation.

[0109] Diffusion models can be categorized into Denoising Diffusion Probabilistic Models (DDP) and Denoising Diffusion Implicit Models (DDIM). In DDDP, the generation process involves a reverse stochastic Markov diffusion process. On the other hand, DDIM uses a deterministic process that ensures the same input produces the same output. Diffusion models can also be characterized by whether noise is added to the image or to image features generated by the encoder (e.g., latent diffusion).

[0110] The diffusion model works by learning to recover data by iteratively adding noise to the data during the forward process and then denoising the data during the reverse process. For example, during training, the diffusion model 800 can take the original image 805 in the pixel space 810 as input and apply an image encoder 815 to transform the original image 805 into original image features 820 in the latent space 825. Then, the forward diffusion process 830 gradually adds noise to the original image features 820 to obtain noise features 835 (also in the latent space 825) at various noise levels.

[0111] Next, the backdiffusion process 840 (e.g., U-Net ANN) progressively removes noise from the noise features 835 at various noise levels to obtain denoised image features 845 in the latent space 825. In some examples, the denoised image features 845 are compared with the original image features 820 at each of the various noise levels, and the parameters of the backdiffusion process 840 of the diffusion model are updated based on this comparison. Finally, the image decoder 850 decodes the denoised image features 845 to obtain the output image 855 in the pixel space 810. In some cases, the output image 855 is created at each of the various noise levels. The output image 855 can be compared with the original image 805 to train the backdiffusion process 840. In some cases, the output image 855 refers to a modified image (e.g., a reference image). Figures 4 to 6 (as described).

[0112] In some cases, the image encoder 815 and image decoder 850 are pre-trained before the back-diffusion process 840 is trained. In some examples, the image encoder 815 and image decoder 850 are jointly trained, or the image encoder 815 and image decoder 850 are jointly fine-tuned with the back-diffusion process 840.

[0113] The backdiffusion process 840 can also be guided based on text cue 860 or another guiding cue (such as an image, layout, style, color, segmentation map, etc.). A text encoder 865 (e.g., a multi-mode encoder) can be used to encode the text cue 860 to obtain guiding features 870 in the guiding space 875. The guiding features 870 can be combined with noise features 835 at one or more layers of the backdiffusion process 840 to ensure that the output image 855 includes what is described by the text cue 860. For example, cross-attention blocks within the backdiffusion process 840 can be used to combine the guiding features 870 with the noise features 835.

[0114] Cross-attention, also known as multi-head attention, is an extension of the attention mechanism used in some ANNs, for example, for NLP tasks. In some cases, cross-attention focuses on multiple parts of an input sequence simultaneously, thereby capturing interactions and dependencies between different elements. In cross-attention, there are two input sequences: a query sequence and a key-value sequence. The query sequence represents the elements that require attention, while the key-value sequence contains the elements to be focused on. In some cases, to compute cross-attention, the cross-attention block transforms each element in the query sequence (e.g., using linear projection) into a "query" representation, and transforms the elements in the key-value sequence into "key" and "value" representations.

[0115] Cross-attention blocks calculate an attention score by measuring the similarity between each query representation and the key representation, where higher similarity indicates more attention being given to the key element. The attention score indicates the importance or relevance of each key element to the corresponding query element.

[0116] Cross-attention blocks are then normalized to obtain attention weights (e.g., using a softmax function), where attention weights determine how much information from each value element is incorporated into the final attention representation. By simultaneously focusing on different parts of the key-value sequence, cross-attention blocks capture relationships and dependencies across the input sequence, enabling machine learning models to understand context and generate more accurate and context-sensitive outputs.

[0117] In some examples, the diffusion model is based on a neural network architecture called U-Net. U-Net takes input features with an initial resolution and an initial number of channels, and uses initial neural network layers (e.g., convolutional network layers) to process the input features to generate intermediate features. The intermediate features are then downsampled using downsampling layers, such that the downsampled features have a resolution smaller than the initial resolution and a number of channels larger than the initial number of channels.

[0118] This process is repeated multiple times, and then reversed. For example, an upsampling process is used to upsample downsampled features to obtain upsampled features. The upsampled features can be combined with intermediate features having the same resolution and number of channels via skip connections. These inputs are then processed using a final neural network layer to produce output features. In some cases, the output features have the same resolution and the same number of channels as the initial resolution.

[0119] In some cases, U-Net employs additional input features to produce conditionally generated outputs. For example, additional input features may include vector representations of input prompts. These additional input features can be combined with intermediate features within the neural network at one or more layers. For instance, cross-attention modules can be used to combine additional input features and intermediate features. (Reference) Figure 7 and Figure 9 Further details about U-Net are described.

[0120] The diffusion process can also be modified based on conditional guidance. In some cases, the user provides text cue words describing what should be included in the generated image (e.g., text cue word 860). In some examples, guidance can be provided in forms other than text (such as via images, sketches, colors, styles, or layouts). The system converts text cue word 860 (or other guidance) into conditional guidance vectors or other multidimensional representations. For example, a transformer model or multimodal encoder can be used to convert text into vectors or a series of vectors. In some cases, the encoder for conditional guidance is trained independently of the diffusion model.

[0121] Initialization involves a noise map containing random noise. The noise map can be in pixel space or latent space. By initializing the image with random noise, different variations of the image, including those described by conditional guidance, can be generated. Then, the diffusion model 800 generates the image based on the noise map and the conditional guidance vector.

[0122] The diffusion process can include both a forward diffusion process 830 for adding noise to an image (e.g., original image 805) or feature (e.g., original image feature 820) in the latent space 825 and a reverse diffusion process 840 for denoising the image (or feature) to obtain a denoised image (e.g., output image 855). The forward diffusion process 830 can be represented as... Furthermore, the reverse diffusion process 840 can be represented as . refer to Figure 11 Further details about the diffusion process are described.

[0123] Both forward diffusion process 830 and backward diffusion process 840 can be used to train the diffusion model 800. In one example, the user initializes the untrained model. Initialization can include defining the model architecture and establishing initial values ​​for the model parameters. In some cases, initialization can include defining hyperparameters such as the number of layers, the resolution and channels of each layer block, the location of skip connections, etc.

[0124] The system then N In each stage, a forward diffusion process 830 is used to add noise to the training image. In some cases, the forward diffusion process 830 is a fixed process in which Gaussian noise is continuously added to the image. In the latent diffusion model, Gaussian noise can be continuously added to features in the latent space 825 (e.g., original image features 820).

[0125] In each stage n From the stage N Initially, the backdiffusion process 840 was used to predict the stage. nThe image or image features at -1. For example, the backdiffusion process 840 can predict the noise added by the forward diffusion process 830 and can remove the predicted noise from the image to obtain the predicted image. In some cases, the original image 805 is predicted at each stage of the training process.

[0126] Training components (e.g., reference) Figure 16 The described training components will be phased n The predicted image (or image feature) at -1 is compared with the actual image (or image feature), such as the stage. n The image at -1 or the original input image. For example, given the observation data. x It is possible to train a diffusion model 800 to minimize the negative log-likelihood of the training data. The variational upper bound is then determined. The trained components are then updated with the parameters of the diffusion model 800 based on this comparison. For example, gradient descent can be used to update the parameters of U-Net. It is also possible to learn the time-dependent parameters of the Gaussian transform. (See reference...) Figure 15 Further details regarding the training of the diffusion model are described.

[0127] Original image 805 is a reference Figure 11 Examples of the corresponding elements described, or including references Figure 11 The description covers various aspects of the corresponding element. The image encoder 815 is a reference. Figure 6 and Figure 16 Examples of the corresponding elements described, or including references Figure 6 and Figure 16 The description covers various aspects of the corresponding element. The forward diffusion process 830 is a reference. Figure 11 Examples of the corresponding elements described, or including references Figure 11 The various aspects of the corresponding element described.

[0128] The reverse diffusion process 840 is a reference. Figure 11 Examples of the corresponding elements described, or including references Figure 11 The description covers various aspects of the corresponding element. Text prompt word 860 is for reference. Figure 7 Examples of the corresponding elements described, or including references Figure 7 The description covers various aspects of the corresponding element. The text encoder 865 is a reference. Figure 6 , Figure 7 and Figure 16 Examples of the corresponding elements described, or including references Figure 6 , Figure 7 and Figure 16 The various aspects of the corresponding element described.

[0129] Figure 9An example of a U-Net 900 architecture according to various aspects of this disclosure is shown. The example shown includes U-Net 900, input features 905, an initial neural network layer 910, intermediate features 915, a downsampling layer 920, downsampled features 925, an upsampling process 930, upsampled features 935, skip connections 940, a final neural network layer 945, and output features 950.

[0130] In some examples, U-Net 900 is the execution reference. Figure 8 Examples of components of the reverse diffusion process 840 of the described diffusion model 800, and including references Figure 16 The architectural elements of the image generation model 1630 described. Figure 9 The U-Net 900 depicted in the image is a reference. Figure 8 Examples of the architectures used in the described reverse diffusion process, or including references. Figure 8 The various aspects of the architecture used within the described reverse diffusion process.

[0131] In some examples, the diffusion model is based on a neural network architecture called U-Net. U-Net 900 takes input features 905 with an initial resolution and an initial number of channels, and uses an initial neural network layer 910 (e.g., a convolutional network layer) to process the input features 905 to produce intermediate features 915. The intermediate features 915 are then downsampled using a downsampling layer 920, such that the downsampled features 925 have a resolution smaller than the initial resolution and a number of channels greater than the initial number of channels.

[0132] This process is repeated multiple times, and then reversed. For example, upsampling process 930 is used to upsample downsampled feature 925 to obtain upsampled feature 935. Upsampled feature 935 can be combined with intermediate feature 915 having the same resolution and number of channels via skip connection 940. These inputs are processed using final neural network layer 945 to produce output feature 950. In some cases, output feature 950 has the same resolution and the same number of channels as the initial resolution.

[0133] In some cases, U-Net 900 employs additional input features to generate conditionally generated outputs. For example, additional input features may include vector representations of input prompts. These additional input features can be combined with intermediate features 915 within the neural network at one or more layers. For instance, a cross-attention module can be used to combine additional input features and intermediate features 915.

[0134] U-Net 900 is a reference. Figure 7 Examples of the corresponding elements described, or including references Figure 7 The description covers various aspects of the corresponding element. Output feature 950 is a reference. Figure 7 Examples of the corresponding elements described, or including references Figure 7 The various aspects of the corresponding element described.

[0135] Figure 10 An example of a diffusion transformer 1000 model according to various aspects of this disclosure is shown. The example shown includes predicted noise 1005, predicted covariance 1010, linear and reshaping layers 1015, normalization layer 1020, (multiple) DiT blocks 1025, patchify operation 1030, embedding 1035, noise latent 1040, time step information 1045, label information 1050, and an implementation of one of the (multiple) DiT blocks 1025 by DiT block 1096. In one aspect, the DiT block 1096 includes a second residual connection 1060, a second scaling operation 1062, a feedforward network 1064, post-normalized second scaling and shifting 1066, a second normalization 1068, a first residual connection 1070, a first scaling operation 1072, self-attention 1074, post-normalized first scaling and shifting 1076, first normalization 1078, an input lexicon 1080, a modulating lexicon 1082, a multilayer perceptron (MLP) 1084, post-normalized first scaling and shifting parameters 1086, a first scaling parameter 1088, post-normalized second scaling and shifting parameters 1090, and a second scaling parameter 1092. In some embodiments, the architecture uses a latent diffusion transformer 1094. In some embodiments, the DiT block 1096 uses the "adaLN-Zero" technique.

[0136] Diffusion Transformer (DiT) is a popular architecture for diffusion models and is designed to be structurally faithful to standard transformer architectures. DiT incorporates the scaling properties of transformer structures. For Denoising Diffusion Probabilistic Models (DDPM) training images (e.g., spatial representations of images), DiT is based on the Visual Transformer (ViT) architecture, which operates on a sequence of patches. DiT processes images by dividing them into patches, converting these patches into tokens, and applying attention mechanisms to model the relationships between different regions of the image. This approach allows the model to capture both local and long-range dependencies in the image generation process.

[0137] In some cases, the input to DiT is a spatial representation. z For a 256×256×3 image, z It has a shape of 32×32×4. The first layer of DiT performs a block operation, where DiT divides the input image into patches and transforms the patches (in the form of spatial input) into dimensions by linearly embedding each patch into the input.d Each dimension in T A sequence of lexical units. After block processing, positional embeddings based on ViT frequency are applied to all input lexical units. In some cases, the number of lexical units created by block processing is... T Patch size hyperparameter p Confirmed. In some cases, T =( I / p ) 2 ,in I It is another shape parameter, therefore p Halving will T This increases the number of operations by four times, which in some cases at least quadruples the total number of gigaflops (Gflops) performed by the converter. In some examples, the change... p It has no effect on downstream parameter counts; for example, parameter counts in downstream layers of DiT are independent of... p Yes. In some examples, p =2, 4, or 8. Various patch sizes, transformer block architectures, and model sizes are implemented.

[0138] Following the block-based operation, an attention mechanism is applied to model the relationships between different regions of the image within one or more DiT blocks. In addition to the noisy image input, the diffusion model sometimes processes additional conditional information, such as the noise time step. t Class tags c Natural language information, etc. The following describes four variants of the transformer block for processing conditional inputs that include both input information and conditional information.

[0139] In some cases, the DiT module in the DiT network is implemented using an Adaptive Layer Normalization (adaLN) block. Following the adaptive normalization layer in a Generative Adversarial Network (GAN) and a traditional diffusion model with a U-Net backbone, in some examples, Adaptive Layer Normalization (adaLN) replaces the standard normalization layer in the transformer block. In adaLN, the system operates from a noise time step... t and class tags c Instead of directly learning γ and β, adaLN regresses the dimensional scaling γ and shift parameters β from the sum of the embedding vectors. adaLN adds relatively few Gflops and is more efficient. Additionally, adaLN is a modulation mechanism that applies the same function to all lexical units.

[0140] In some cases, adaLN-Zero blocks, which utilize zero initialization techniques, are used to implement DiT blocks in DiT networks. In ResNet, each residual block is initialized with an identity function. This is beneficial. In some examples, zero-initializing the final batch normalization scaling factor γ in each block accelerates large-scale training in supervised learning settings. The U-Net-based diffusion model uses a similar initialization strategy, zero-initializing the final convolutional layer in each block before the residual connections. The adaLN-Zero block is modified from the adaLN block using a similar zero-initialization technique. In addition to the scaling and γ and shift parameters β in the regression dimension, the system also regresses the scaling parameter αs in the regression dimension, which is applied immediately before the residual connections within the DiT block. The network initializes the multilayer perceptron (MLP) to output a zero vector for all αs; this initializes the entire DiT block to the identity function. As with the adaLN block, adaLNZero adds negligible Gflops to the model.

[0141] In some cases, context-based modulation is used to implement DiT blocks in DiT networks, where t and c The vector embeddings are appended as two additional terms in the input sequence, and after the final block, the network removes two modulating terms from the sequence.

[0142] In some cases, DiT blocks in DiT networks include cross-attention blocks. DiT networks will... t and c The embeddings are concatenated into a sequence of length 2, which is separated from the image word sequence. The transformer block is modified to include an additional multi-head cross-attention layer after the multi-head self-attention block.

[0143] In some cases, DiT networks include N A sequence of DiT blocks, each block having a hidden dimension size of [size missing]. d Operation at the point. Following ViT, DiT networks use a standard converter configuration, which is... N , d And attention heads are used for joint scaling. In some examples, small (S), basic (B), large (L), and extra-large (XL) variants of the model size are implemented. Small or basic model sizes have N =12 layers of DiT blocks, large model size has 24 layers of DiT blocks. Extra large model size has 28 layers of DiT blocks.

[0144] Following the final DiT block, the DiT network decodes the image word sequence into an output noise prediction and an output diagonal covariance prediction. Both outputs have the same shape as the original spatial input. Decoding is performed using a standard linear decoder, where the final normalization layer (or an adaptive normalization layer if the DiT block is an adaLN block) linearly decodes each word into... p × p ×2C The tensor, in which C It is the number of channels in the spatial input of the DiT network, and p This is the patch size hyperparameter. Finally, the decoded tokens are rearranged to their original spatial layout to obtain the predicted noise and covariance.

[0145] In some cases, the diffusion transformer model employs a latent diffusion transformer 1094. The diffusion transformer model processes a noisy latent 1040, which can be a noisy version of the input image encoded in the latent space. A block operation 1030 divides the noisy latent into a sequence of patches that are processed as terms. A term is a vector representation of each patch of the image in the latent space and is adjusted through an attention process. Each term also receives time step information 1045 and label information 1050, and an embedding 1035 encodes the current denoising time step and class label as conditional information. In some cases, the embedding 1035 is referred to as a conditional embedding or conditional information embedding. In some cases, at the block operation 1030, a positional embedding encoding the spatial location of each term in the image is applied to the block's input terms. In some examples, the positional embedding is a positional embedding based on ViT frequencies. The input lexical unit 1080 generated by the block operation 1030 and the conditioning lexical unit 1082 generated by the embedding 1035 are processed by N DiT blocks 1025, where N can be 102, 24, or 28. Other values ​​of N can be used. In some cases, the conditioning lexical unit refers to the lexical unit generated based on the embedding 1035, the encoding time step information 1045, and the tag information 1050.

[0146] Each DiT block in the plurality of DiT blocks 1025 includes multiple processing stages. DiT block 1096 illustrates an embodiment of one of the blocks in the plurality of DiT blocks 1025. In some embodiments, DiT block 1096 is an example of an adaLN-Zero block, or includes aspects of an adaLN-Zero block. In some cases, input lexicon 1080 interacts with conditioning lexicon 1082 through multiple attention mechanisms. Specifically, after applying a first normalization 1078 to the input lexicon and applying MLP 1084 to the conditional lexicon, MLP 1084 generates or updates post-normalized first scaling and shift parameters 1086, which are denoted as γ1, β1 for post-normalized first scaling and shift 1076 to scale and shift the output of the first normalization 1078 accordingly. This allows input and conditional information to interact when the normalized input lexical obtained from the first normalization 1078 is scaled and shifted at the post-normalization first scaling and shift 1076 using conditional information carried in at least γ1 and β1. Self-attention 1074 allows scaled and shifted normalized input lexical, i.e., the output from the post-normalization first scaling and shift 1076, to attend to each other. MLP 1084 also generates or updates a first scaling parameter 1088, denoted as α1, for scaling the output of self-attention 1074 (e.g., multi-head self-attention) by the first scaling operation 1072, thereby further interacting with the input and conditional information. Input lexical 1080 is then added to the output of the first scaling operation 1072 at the first residual connection 1070. In some examples, α1 has an initial value of 0, and DiT block 1096 is initialized to an identity function.

[0147] A similar process is performed in the second half of DiT block 1096. MLP 1084 generates or updates post-normalized second scaling and shift parameters 1090, denoted as γ2, β2, for post-normalized second scaling and shifting 1066 to correspondingly scale and shift the output of second normalization 1068. This allows further interaction between the input and conditional information when the output from second normalization 1068 is scaled and shifted using conditional information carried at least in γ2, β2. Feedforward network 1064 then processes the scaled and shifted output from post-normalized second scaling and shifting 1066. MLP 1084 also generates or updates a second scaling parameter 1092, denoted as α2, for scaling the output of feedforward network 1064 in second scaling operation 1062, thereby further interacting with the input and conditional information. In some cases, feedforward network 1064 is a pointwise feedforward network. The output from the first residual connection 1070 is then added to the output of the second scaling operation 1062 at the second residual connection 1060, and the result is the final output of DiT block 1096. In some examples, α2 has an initial value of 0, and DiT block 1096 is initialized to the identity function. This process is repeated for each DiT block in the sequence.

[0148] After processing through all (multiple) DiT blocks 1025, the output undergoes a normalization layer 1020, followed by a linearization and reshaping layer 1015. The final output is predicted noise 1005 and predicted covariance 1010, where predicted noise 1005 represents the model's prediction of the noise added to initially create the noise potential 1040, and predicted covariance 1010 represents the model's prediction of the covariance. Predicted noise 1005 is removed from the noise potential 1040 at each diffusion time step, and predicted covariance can influence how noise is removed or resampled during inversion or denoising. At the end of the denoising scheme, the latent samples are decoded to generate a synthetic image in pixel space. diffusion process

[0149] Figure 11 Examples of diffusion processes 1100 according to various aspects of this disclosure are shown. The examples shown include diffusion process 1100, forward diffusion process 1105, reverse diffusion process 1110, noisy image 1115, first intermediate image 1120, second intermediate image 1125, and original image 1130.

[0150] The diffusion process 1100 can include a forward diffusion process 1105, which is used to add noise to the original image 1130 in the latent space (e.g., reference). Figure 8 The original image 805 described or features (e.g., reference) Figure 8The original image features 820 are described. In some aspects, the diffusion process 1100 includes a reverse diffusion process 1110, which is used to denoise the noisy image 1115 (or image features) to obtain a denoised image (or the original image 1130). The forward diffusion process 1105 can be represented as... And the reverse diffusion process 1110 can be represented as In some cases, a forward diffusion process 1105 is used during training to generate images with continuously large noise, and the neural network is trained to perform a backward diffusion process 1110 (e.g., to continuously remove noise).

[0151] In response to potential diffusion models (e.g., reference) Figure 8 In the example forward diffusion process 1105 of the diffusion model (described in 800), the diffusion model uses a Markov chain to map the observed variables. x 0 (in pixel space or latent space) to obtain intermediate variables x 1, …, x T Markov chains progressively add Gaussian noise to the data to obtain an approximate posterior as latent variables are passed through a neural network (such as U-Net). ,in x 1, …, x T Having the same dimension x 0.

[0152] A neural network can be trained to perform the backdiffusion process 1110. During the backdiffusion process 1110, the diffusion model uses noisy data... x T (Such as noisy image 1115) begins, and the data is denoised to obtain In each step t At -1, the reverse diffusion process 1110 is adopted. x t (such as the first intermediate image 1120) and t As input. Here, t This represents the steps in the transformation sequence associated with different noise levels. The backdiffusion process 1110 iteratively outputs... x t-1 (such as the second intermediate image 1125), until x T Restore back x 0 (Original image 1130). The back diffusion process 1110 can be represented as:

[0153] The joint probability of a sequence of samples in a Markov chain can be written as the product of the conditional probability and the marginal probability: When the reverse diffusion process 1110 uses the output of the forward diffusion process 1105 and a pure noise sample as input, It is a pure noise distribution, and This represents the Gaussian transformed sequence corresponding to the sequence to which Gaussian noise has been added to the sample.

[0154] During the interference time, it is possible to extract the observed data in the pixel space. x 0 Mapped into the latent space as input, and the generated data As output, it is mapped back from the latent space to the pixel space. In some examples, x 0 The latent variable represents the original input image with low image quality. x 1, …, x T Represents a noisy image, and This indicates a generated image with high image quality.

[0155] Forward diffusion process 1105 is a reference. Figure 8 Examples of the corresponding elements described, or including references Figure 8 The description covers various aspects of the corresponding element. The back diffusion process 1110 is a reference. Figure 8 Examples of the corresponding elements described, or including references Figure 8 The description covers various aspects of the corresponding element. Original image 1130 is for reference. Figure 8 Examples of the corresponding elements described, or including references Figure 8 The various aspects of the corresponding element described. Training and assessment

[0156] exist Figures 12 to 15 Among them, a method, apparatus, non-transitory computer-readable medium, and system for image processing include: obtaining a training set including training images, modification cue words describing changes to the training images, and modified training images depicting changes to the training images; encoding the modification cue words using a text encoder to obtain text embeddings, wherein the text embeddings represent modifications in an embedding space; and training an image generation model using the training set and the text embeddings to generate synthetic images based on the modification cue words.

[0157] In some aspects, the image generation model is trained to preserve elements of the training images that are not described by the modified cue words. In other aspects, the training set includes masked inputs indicating regions of the training images, wherein the image generation model is trained to generate synthetic images based on the masked inputs.

[0158] Examples of the method, apparatus, non-transitory computer-readable medium, and system also include generating a first predicted image based on captions for a training image. Examples of the method, apparatus, non-transitory computer-readable medium, and system also include generating a second predicted image based on modified prompts.

[0159] Examples of the method, apparatus, non-transitory computer-readable medium, and system further include generating a first image feature based on training images. Examples of the method, apparatus, non-transitory computer-readable medium, and system further include generating a second image feature based on modified prompt words. Examples of the method, apparatus, non-transitory computer-readable medium, and system further include combining the first image feature and the second image feature to obtain a combined image feature.

[0160] Examples of the method, apparatus, non-transitory computer-readable medium, and system also include computing a diffusion loss, based on which the image generation model is trained. Examples of the method, apparatus, non-transitory computer-readable medium, and system also include modifying the color of the training image to obtain a modified training image.

[0161] Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include captions identifying training images. Some examples also include generating training cue words based on the captions. Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include segmenting a first predicted image to obtain a mask indicating the location of an object, wherein a second predicted image is based on the mask.

[0162] Examples of the method, apparatus, non-transitory computer-readable medium, and system include generating a first predicted image based on captions. Examples also include generating a second predicted image based on training prompts. Examples further include calculating a loss function based on the first and second predicted images. Examples also include updating the parameters of an image generation model based on the loss function.

[0163] Figure 12 Examples of a method 1200 for training a machine learning model according to aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor with functional elements that execute sets of code to control means. Additionally or alternatively, dedicated hardware is used to perform certain processes. Generally, these operations are performed according to the methods and processes described according to aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0164] At operation 1205, the system obtains a training set including training images, modification prompts describing changes to the training images, and modified training images depicting the changes to the training images. In some cases, this step involves referencing... Figure 16 The training components described, or those that can be found in the reference... Figure 16 The described training components are used to perform this. (See reference.) Figure 13 Further details regarding obtaining the training set are described.

[0165] At operation 1210, the system uses a text encoder to encode the modification prompt word to obtain a text embedding, where the text embedding represents the modification in the embedding space. In some cases, this step involves referencing... Figure 16 The training components described, or those that can be found in the reference... Figure 16 The described training components are used to perform this step. In some cases, this step involves referencing... Figure 16 The text encoder described, or which may be referenced Figure 16 The text encoder described is used for execution. (Reference) Figure 13 Further details regarding the training of the image generation model are described.

[0166] At operation 1215, the system uses the training set and text embeddings to train an image generation model to generate synthetic images based on modified prompts. In some cases, this step involves referencing... Figure 16 The training components described, or those that can be found in the reference... Figure 16 The described training components are used to perform this. (See reference.) Figure 13 Further details regarding the training of the image generation model are described.

[0167] Figure 13 Examples of training data generated according to various aspects of this disclosure are shown. The examples shown include a training system 1300, an original dataset 1305, text-image pairs 1310, text captions 1315, training images 1320, a trained language generation model 1325, trained cue words 1330, a trained image generation model 1335, a first predicted image 1340, a second predicted image 1345, a segmentation model 1350, and a training mask 1355. In one aspect, the training system 1300 includes training the language generation model 1325, training the image generation model 1335, and the segmentation model 1350.

[0168] refer to Figure 13 Training components (e.g., reference) Figure 16The described training component uses a training dataset obtained from the original dataset 1305 to train an image generation model (e.g., train image generation model 1335). For example, the original dataset 1305 includes text-image pairs 1310, where each text-image pair 1310 includes a training image 1320 and a text caption 1315 describing one or more elements of the training image 1320. For example, the text caption 1315 states "steak cooking on a grill".

[0169] The language generation model 1325 generates training prompts 1330 based on the text caption 1315 by modifying elements from the text caption. For example, the training prompt 1330 states "fish cooking on a grill." Alternatively, the language generation model 1325 may replace "steak" from the text caption 1315 with "fish" from the training prompt 1330. In some cases, the language generation model is a pre-trained model, such as Llama2.

[0170] In some embodiments, the trained image generation model 1335 generates predicted images based on cue words. For example, the trained image generation model 1335 generates a first predicted image 1340 based on text captions 1315, the first predicted image 1340 depicting a steak cooking on a grill. For example, the trained image generation model 1335 generates a second predicted image 1345 based on training cue words 1330, the second predicted image 1345 depicting fish cooking on a grill. In some embodiments, the first predicted image 1340 is provided to the segmentation model 1350 to generate a training mask 1355. In some cases, the training mask 1355 is provided to the trained image generation model 1335 to guide the image generation process to generate the second predicted image 1345.

[0171] According to some embodiments, the training data includes prompts, input images, and output images. In some cases, the training data also includes a mask. In some cases, the training data includes reference inputs used to further train the image generation model. For example, the reference inputs include fine-grained color images, texture images, and style images.

[0172] In some embodiments, training data is filtered to generate a training set used to train the image generation model. In some cases, for example, a Large Language Model (LLM) or a language generation model is used to generate the training set. For example, 1M of labeled data is used to fine-tune a pre-trained language generation model (e.g., an LLAVA-based model). In some aspects, the language generation model takes the original image, the modified image, and the text instruction as input and generates an output that evaluates whether paired images (e.g., the original image and the modified image) are aligned with the text instruction while preserving the identifiers.

[0173] In some embodiments, the training component uses a language generation model to generate instructions (e.g., training cue word 1330). In some cases, the language generation model is able to identify and extract entities from text captions. In some cases, the language generation model generates variations of the training cue word. According to some embodiments, masking control is applied to the image generation model to enhance image quality. For example, a mask is applied to each denoising step in the image generation process to ensure local editing.

[0174] According to some embodiments, the image generation model is fine-tuned or refined to perform identity preservation. For example, the system interpolates image features (e.g., DINO features) of the input image and the modified image to enhance identity preservation. For color modification, the model is trained using ControlNet and an image generation model (e.g., ClioMD). For example, a grayscale input image of the mask is used as input. In some cases, edge maps are used to further refine the image generation model. In some cases, the region outside the mask includes raw pixels from the input image, and the region inside the mask is grayscale. Accordingly, the system ensures that color modification is performed within the region indicated by the mask. According to some embodiments, the system performs texture-based editing, where texture input is converted into synthetic data.

[0175] Depending on the aspect, the system is jointly trained with multiple tasks including inpainting, expansion, editing, segmentation, depth estimation, normal vector estimation, colorization, and low-level vision. For example, inpainting is a technique that fills in missing or damaged portions of an image by predicting content based on surrounding pixels. Inpainting is used for tasks such as restoring damaged photos or removing objects from a scene. For example, expansion involves extending an image beyond its original boundaries to generate new content aligned with the existing scene. For example, editing tasks involve modifying specific regions of an image, such as changing colors, textures, or even object placement, while maintaining the consistency of the original content.

[0176] For example, segmentation is the process of dividing an image into one or more regions or objects based on certain features such as color, texture, object, semantics, or entity. For example, depth estimation predicts the distance of an object from the viewpoint in the scene. For example, colorization involves adding color to a grayscale image by predicting the appropriate hue for each pixel. For example, low-level vision involves extracting features from an image, such as edges, texture, and gradients.

[0177] Figure 14 An example flowchart of a step-by-step procedure in an exemplary implementation of an algorithm according to various aspects of this disclosure, which is capable of performing operations for training a machine learning model, is shown. In some embodiments, procedure 1400 describes as referenced... Figure 16The described operation of the training component 1635 for configuring the image generation model 1630 is described. Program 1400 provides one or more examples of generating training data, using the training data to train a machine learning model, and using the trained machine learning model to perform a task.

[0178] To begin this example, the machine learning system collects training data (block 1402) to serve as the basis for training a machine learning model that defines what is being modeled. Training data is collected by the machine learning system from a variety of sources. Examples of training data sources include public datasets, service provider system platforms that demonstrate application programming interfaces (APIs) (e.g., social media platforms), user data collection systems (e.g., digital surveys and online crowdsourcing systems), etc. Training data collection may also include: data augmentation and synthetic data generation techniques to expand and diversify the available training data, balancing techniques to balance multiple positive and negative examples, etc.

[0179] The machine learning system can also be configured to identify features relevant to the type of task for which a machine learning model is to be trained. Examples of tasks include classification, natural language processing, generative artificial intelligence, recommendation engines, reinforcement learning, clustering, etc. To this end, the machine learning system collects training data based on the identified features and / or filters the collected training data based on the identified features. The training data is then used to train the machine learning model.

[0180] To train the machine learning model in the illustrated example, the machine learning model is first initialized (block 1406). Initialization of the machine learning model includes selecting the model architecture to be trained (block 1408). Examples of model architectures include neural networks, convolutional neural networks (CNNs), long short-term memory (LSTM) neural networks, generative adversarial networks (GANs), decision trees, support vector machines, linear regression, logistic regression, Bayesian networks, random forest learning, dimensionality reduction algorithms, boosting algorithms, deep learning neural networks, and the U-Net architecture.

[0181] A loss function is also selected (block 1410). The loss function is used to measure the difference between the output of the machine learning model (e.g., model prediction) and the target value to be used to train the machine learning model (e.g., as expressed by the training data). Additionally, an optimization algorithm (block 1412) is selected to be used in conjunction with the loss function to optimize the parameters of the machine learning model during training; examples include gradient descent, stochastic gradient descent (SGD), etc.

[0182] Initializing the machine learning model also includes setting initial values ​​for the model (block 1416), examples of which include initializing node weights and biases to improve the efficiency of training and computational resource consumption as part of the training process. Hyperparameters used to control the training of the machine learning model are also set (block 1414), examples of which include regularization parameters, model parameters (e.g., the number of layers in the neural network), learning rate, batch size selected from the training data, etc. Various techniques are used to set hyperparameters, including using randomization techniques, using heuristics learned from other training scenarios, etc.

[0183] The machine learning system then uses the training data to train the machine learning model (block 1418). A machine learning model refers to a computer representation that can be tuned (e.g., trained and retrained) based on inputs of training data to approximate an unknown function. Specifically, the term machine learning model can encompass models that learn from and predict known data by utilizing algorithms (e.g., using the model architecture described above) to learn and relearn by analyzing training data to generate outputs that reflect the patterns and properties expressed by the training data.

[0184] Examples of training types include supervised learning using labeled data, unsupervised learning involving finding underlying structures or patterns within the training data, reinforcement learning based on optimization functions (e.g., rewards and / or penalties), and using nodes as part of "deep learning." For example, a machine learning model can be configured as multiple nodes that collectively form multiple layers. These layers can be configured to include an input layer, an output layer, and one or more hidden layers. Computation is performed by the nodes within the layers through hidden states, via a system of weighted connections "learned" during training, for example, by optimizing the performance of the machine learning model to perform the associated task using a chosen loss function and backpropagation.

[0185] As part of training the machine learning model, a stopping criterion used to verify whether the machine learning model has been met is determined (decision block 1420). This stopping criterion can be used to reduce overfitting of the machine learning model, reduce computational resource consumption, and improve the machine learning model's ability to handle unseen data not included as examples in the training data. Examples of stopping criteria include, but are not limited to, a predefined number of epochs, verifying loss stability, reaching a performance improvement threshold, whether a threshold accuracy level has been met, or based on performance metrics such as precision and recall. If the stopping criterion has not yet been met ("No" from decision block 1420), then in this example, procedure 1400 continues training the machine learning model using the training data (block 1418).

[0186] If the stopping criterion is met ("Yes" from decision block 1420), then the trained machine learning model is used to generate an output based on subsequent data (block 1422). For example, the trained machine learning model is trained to perform the task as described above, and therefore, once trained, is configured to perform the task based on subsequent data received as input and processed by the machine learning model.

[0187] Figure 15 Examples of a method 1500 for training a diffusion model according to aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor with functional elements that execute a set of code to control the means. Additionally or alternatively, dedicated hardware is used to perform certain processes. Generally, these operations are performed according to the methods and processes described according to aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0188] In some embodiments, method 1500 is described as referenced. Figure 16 The described operation is for the training component 1635 used to train the image generation model 1630. Method 1500 represents the operation for training as referenced above. Figure 11 Examples of the described reverse diffusion process. In some examples, these operations are performed by a system that includes a processor comprising functional elements that execute sets of code to control the device, such as... Figure 16 The image generation model described in [the document / document].

[0189] At operation 1505, the system initializes the untrained model. In some cases, this step involves referencing... Figure 16 The training components described, or those that can be found in the reference... Figure 16 The training components described are used to perform this process. Initialization can include defining the model's architecture and establishing initial values ​​for the model parameters. In some cases, initialization can include defining hyperparameters such as the number of layers, the resolution and channels of each layer block, and the location of skip connections.

[0190] At operation 1510, the system uses a forward diffusion process to add noise to the medium term in N stages. In some cases, this step involves a reference... Figure 16 The training components described, or those that can be found in the reference... Figure 16 The training components described are used to perform this process. In some cases, for example, the media item is the training image. In other cases, the forward diffusion process is a fixed process in which Gaussian noise is continuously added to the media item (such as the original image). In the latent diffusion model, Gaussian noise can be continuously added to features in the latent space.

[0191] At operation 1515, the system predicts the media item for stage n-1 at each stage n starting from stage N. In some cases, this step involves... Figure 16 The training components described, or those that can be found in the reference... Figure 16 The training components described are used to perform this. In some cases, the media item is a synthetic image generated using an image generation model. For example, the backdiffusion process can predict the noise added by the forward diffusion process and remove the predicted noise from the noise input to obtain the predicted output. In some cases, the original media item is predicted at each stage of the training process.

[0192] At operation 1520, the system compares the predicted media item (or feature) at stage n-1 with the media at stage n-1. In some cases, for example, the system compares the synthesized image (or predicted image features) at stage n-1 with the ground truth image (or ground truth features) at stage n-1. In some cases, this step involves referencing... Figure 16 The training components described, or those that can be found in the reference... Figure 16 The described training components are used to execute this. For example, given the observed data... x A diffusion model can be trained to minimize the negative log-likelihood of the training data. The variational upper bound.

[0193] At operation 1525, the system updates the model's parameters based on comparisons. In some cases, this step involves referencing... Figure 16 The training components described, or those that can be found in the reference... Figure 16 The described training components are used to perform this. For example, gradient descent can be used to update the parameters of U-Net. It is also possible to learn the time-dependent parameters of the Gaussian transform. System device

[0194] Figure 16 Examples of an image processing apparatus 1600 according to various aspects of the present disclosure are shown. The illustrated examples include an image processing apparatus 1600, a processor unit 1605, an I / O module 1610, a memory unit 1615, and a training component 1635. In one aspect, the memory unit 1615 includes a text encoder 1620, an image encoder 1625, and an image generation model 1630.

[0195] According to some embodiments of this disclosure, image processing apparatus 1600 includes a computer-implemented artificial neural network (ANN). An ANN is a hardware or software component comprising multiple connected nodes (e.g., artificial neurons) that roughly correspond to neurons in the human brain. Each connection or edge sends a signal from one node to another (such as a physical synapse in the brain). When a node receives a signal, it processes the signal and then transmits the processed signal to other connected nodes. In some cases, the signals between nodes include real numbers, and the output of each node is computed by adding its inputs to a function. In some examples, nodes may use other mathematical algorithms (e.g., selecting the maximum value from the inputs as the output) or any other suitable algorithm for activating nodes to determine the output. Each node and edge is associated with one or more node weights that determine how the signal is processed and transmitted. Image processing apparatus 1600 is a reference. Figure 1 Examples of the corresponding elements described, or including references Figure 1 The various aspects of the corresponding element described.

[0196] Processor unit 1605 is an intelligent hardware device (e.g., a general-purpose processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof). In some cases, processor unit 1605 is configured to use a memory controller to operate a memory array. In other cases, the memory controller is integrated into the processor. In some cases, processor unit 1605 is configured to execute computer-readable instructions stored in memory to perform various functions. In some embodiments, processor unit 1605 includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing. Processor unit 1605 is a reference. Figure 17 Examples of processors described, or including references Figure 17 The processor described in terms of various aspects.

[0197] I / O module 1610 (e.g., input / output interface) may include an I / O controller. The I / O controller can manage input and output signals to the device. The I / O controller can also manage peripheral devices not integrated into the device. In some cases, the I / O controller may represent a physical connection or port to an external peripheral device. In some cases, the I / O controller may utilize an operating system (such as iOS®, ANDROID®, MS-DOS®, MS-WINDOWS®, OS / 2®, UNIX®, LINUX®, or other known operating systems). In other cases, the I / O controller may represent, or interact with, a modem, keyboard, mouse, touchscreen, or similar device. In some cases, the I / O controller may be implemented as part of a processor. In some cases, a user may interact with the device via the I / O controller or hardware components controlled by the I / O controller.

[0198] In some examples, I / O module 1610 includes a user interface. The user interface enables a user to interact with the device. In some embodiments, the user interface may include an audio device (such as an external speaker system), an external display device (such as a display screen), or an input device (e.g., a remote control device that interfaces directly with the user interface or via an I / O controller module). In some cases, the user interface may be a graphical user interface (GUI). In some examples, the communication interface operates at the boundary between the communication entity and the channel, and may also record and process communications. Communication interfaces are provided herein to enable a processing system to be coupled to a transceiver (e.g., a transmitter and / or receiver). In some examples, the transceiver is configured to transmit (or send) and receive signals for the communication device via an antenna. I / O module 1610 is a reference. Figure 17 Examples of the described I / O interfaces, or including references Figure 17 The various aspects of the described I / O interface.

[0199] Examples of memory unit 1615 include random access memory (RAM), read-only memory (ROM), or hard disk. Examples of memory unit 1615 include solid-state memory and hard disk drives. In some examples, memory unit 1615 is used to store computer-readable, computer-executable software including instructions that, when executed, cause the processor to perform the various functions described herein.

[0200] In some cases, among others, memory cell 1615 includes a basic input / output system (BIOS) that controls basic hardware or software operations, such as interaction with peripheral components or devices. In some cases, the memory controller operates memory cells. For example, the memory controller can include a row decoder, a column decoder, or both. In some cases, the memory cells within memory cell 1615 store information in the form of logical states.

[0201] In one aspect, memory cell 1615 includes a machine learning model. In another aspect, the machine learning model includes a text encoder 1620, an image encoder 1625, and an image generation model 1630. Memory cell 1615 is a reference. Figure 17 Examples of the described memory subsystems, or including references Figure 17 The described aspects of the memory subsystem.

[0202] In some cases, a machine learning model is a computational algorithm, model, or system designed to recognize patterns, make predictions, or perform a specific task (e.g., image processing) without being explicitly programmed. According to some aspects, the machine learning model is implemented as software, firmware, one or more hardware circuits, or a combination thereof, stored in memory unit 1615 and executable by processor unit 1605.

[0203] According to some embodiments of this disclosure, machine learning models include ANNs, which are hardware or software components comprising multiple connected nodes (e.g., artificial neurons) that roughly correspond to neurons in the human brain. Each connection or edge sends a signal from one node to another (such as a physical synapse in the brain). When a node receives a signal, it processes the signal and then sends the processed signal to other connected nodes. In some cases, the signals between nodes consist of real numbers, and the output of each node is computed by a function that adds its inputs. In some examples, nodes may use other mathematical algorithms (e.g., selecting the maximum value from the input as the output) or any other suitable algorithm for activating nodes to determine the output. Each node and edge is associated with one or more node weights that determine how the signal is processed and sent.

[0204] During training, the weights of one or more nodes are adjusted to increase the accuracy of the results (e.g., by minimizing a loss function that corresponds in some way to the difference between the current result and the target result). Edge weights increase or decrease the strength of the signal transmitted between nodes. In some cases, nodes have a threshold below which no signal is transmitted. In some examples, nodes are clustered into layers. Different layers perform different transformations on their corresponding inputs. The initial layer is called the input layer, and the last layer is called the output layer. In some cases, the signal passes through certain layers multiple times.

[0205] According to some embodiments, machine learning models include computer-implemented convolutional neural networks (CNNs). CNNs are a class of neural networks commonly used in computer vision or image classification systems. In some cases, CNNs enable the processing of digital images with minimal preprocessing. CNNs can be characterized by using convolutional (or cross-correlation) hidden layers. These layers apply a convolution operation to the input before passing the resulting signal to the next layer. Each convolutional node can process data for a finite domain (e.g., the receptive domain) of the input. During the forward pass of the CNN, filters at each layer can be convolved over the input volume, thereby computing the dot product between the filter and the input. During the training process, filters can be modified such that they activate when they detect specific features within the input.

[0206] In one aspect, a machine learning model includes machine learning parameters. Machine learning parameters (also called model parameters or weights) are variables that provide the behavior and characteristics of a machine learning model. Machine learning parameters can be learned or estimated from training data and are used to make predictions or perform tasks based on learned patterns and relationships in the data.

[0207] During training, machine learning parameters are tuned to minimize the loss function or maximize the performance metric. The goal of the training process is to find the optimal values ​​of the parameters that allow the machine learning model to make accurate predictions or perform well on a given task.

[0208] For example, during training, the algorithm adjusts its machine learning parameters using optimization techniques such as gradient descent, stochastic gradient descent, or other optimization algorithms to minimize the error or loss between the predicted output and the actual target. Once the machine learning parameters have been learned from the training data, they are used to make predictions on new, unseen data.

[0209] According to some embodiments, machine learning models include computer-implemented recurrent neural networks (RNNs). RNNs are a type of ANN where connections between nodes form a directed graph along an ordered (e.g., temporal) sequence. This allows RNNs to model temporal dynamics (such as predicting what element should appear next in a sequence). Therefore, RNNs are suitable for tasks involving ordered sequences (such as text recognition, where words are ordered within a sentence). In some cases, RNNs include one or more finite spiking recurrent networks (characterized by nodes forming a directed acyclic graph), one or more infinite spiking recurrent networks (characterized by nodes forming a directed cyclic graph), or combinations thereof.

[0210] According to some embodiments, the machine learning model includes a transformer (or transformer model, or transformer network), where a transformer is a neural network model for natural language processing tasks. A transformer network uses an encoder and a decoder to transform one sequence into another. The encoder and decoder include modules that can be stacked on top of each other multiple times. The modules include multi-head attention and feedforward layers. The input and output (target sentence) are first embedded into an n-dimensional space. Positional encodings of different words (e.g., assigning relative positions to each word / part of the sequence, as the sequence depends on the order of its elements) are added to the embedding representation (an n-dimensional vector) of each word. In some examples, the transformer network includes an attention mechanism where attention examines the input sequence and determines at each step which other parts of the sequence are important. The attention mechanism involves a query, a key, and a value, represented by Q, K, and V, respectively. Q is a matrix containing the query (a vector representation of a word in the sequence), K is the key (a vector representation of a word in the sequence), and V is the value, which is also a vector representation of a word in the sequence. For the encoder and decoder, and the multi-head attention module, V consists of the same sequence of words as Q. However, for attention modules that consider both encoder and decoder sequences, V differs from the sequence represented by Q. In some cases, the values ​​in V are multiplied by some attention weights a and summed.

[0211] In machine learning, attention mechanisms (e.g., implemented in one or more ANNs) are methods for assigning different levels of importance to different elements of the input. Computing attention can involve three basic steps. First, the similarity between the query and key vectors obtained from the input is calculated to generate attention weights. Similarity functions used for this process can include dot products, concatenation, detectors, etc. Next, the attention weights are normalized using a softmax function. Finally, the attention weights are weighted together with their corresponding values. In the context of attention networks, keys and values ​​are vectors or matrices used to represent the input data. Keys are used to determine which parts of the input the attention mechanism should focus on, while values ​​represent the actual data being processed.

[0212] Attention mechanisms are a key component in some ANN architectures, particularly those used in Natural Language Processing (NLP) and sequence-to-sequence tasks, allowing ANNs to focus on different parts of the input sequence when making predictions or generating output. Some sequence models, such as RNNs, process the input sequence sequentially, thus preserving internal hidden states that capture information from previous steps. However, in some cases, this sequential processing makes it difficult to capture long-range dependencies or focus on specific parts of the input sequence.

[0213] Attention mechanisms address these difficulties by enabling ANNs to selectively focus on different parts of the input sequence, assigning different degrees of importance or attention to each part. Attention mechanisms achieve selective focusing by considering the relevance of each input element to the current state of the ANN.

[0214] The term "self-attention" refers to a machine learning model in which representations of the input interact with each other to determine attention weights for the input. Because the attention weights are determined at least in part by the input itself, self-attention can be distinguished from other attention models.

[0215] According to some aspects, the text encoder 1620 is implemented as software, firmware, one or more hardware circuits, or a combination thereof, stored in memory unit 1615 and executable by processor unit 1605. According to some aspects, the text encoder 1620 generates a text embedding based on a modification cue word, wherein the text embedding represents a modification in the embedding space. According to some aspects, the text encoder 1620 encodes the modification cue word to obtain the text embedding, wherein the text embedding represents a modification in the embedding space. The text encoder 1620 is a reference. Figures 6 to 8 Examples of the corresponding elements described, or including references Figures 6 to 8 The various aspects of the corresponding element described.

[0216] According to some aspects, the image encoder 1625 is implemented as software, firmware, one or more hardware circuits, or a combination thereof, stored in memory unit 1615 and executable by processor unit 1605. According to some aspects, the image encoder 1625 receives a reference input. In some examples, the image encoder 1625 generates a reference embedding based on the reference input, wherein a modified image is generated based on the reference embedding. In some examples, the image encoder 1625 generates a first image feature based on the input image. In some examples, the image encoder 1625 generates a second image feature based on a text embedding. The image encoder 1625 is a reference... Figure 6 and Figure 8 Examples of the corresponding elements described, or including references Figure 6 and Figure 8 The various aspects of the corresponding element described.

[0217] According to some aspects, the image generation model 1630 is implemented as software, firmware, one or more hardware circuits, or a combination thereof, stored in memory unit 1615 and executable by processor unit 1605. According to some aspects, the image generation model 1630 receives an input image and a modification prompt, wherein the input image depicts an object having a first attribute, and the modification prompt describes a modification from the first attribute to a second attribute different from the first attribute. In some examples, the image generation model 1630 generates a modified image based on the input image and a text embedding, wherein the modified image depicts an object having a second attribute. In some aspects, the modified image retains a third attribute of the object.

[0218] In some examples, image generation model 1630 receives a mask input indicating regions of the input image, wherein a modified image is generated based on the mask input. In some examples, image generation model 1630 receives a noisy input. In some examples, image generation model 1630 denoises the noisy input based on text embedding to obtain the modified image. In some examples, image generation model 1630 modifies the colors of the input image to obtain a modified input image, wherein the modified image is generated based on the modified input image.

[0219] In some examples, the image generation model 1630 combines first and second image features to obtain combined image features, wherein a modified image is generated based on the combined image features. In some aspects, a training set is used to train the image generation model 1630 to edit images; this training set includes training images depicting objects with a first attribute and training cue words describing modifications from the first attribute to the second attribute. In some aspects, the image generation model 1630 is trained using a first predicted image generated based on captions and a second predicted image generated based on changes to the captions.

[0220] In some aspects, image generation model 1630 includes a diffusion model. In some aspects, image generation model 1630 includes a U-Net architecture. In some aspects, image generation model 1630 includes a diffusion transformer. Image generation model 1630 is a reference. Figure 6 Examples of the corresponding elements described, or including references Figure 6 The various aspects of the corresponding element described. computing devices

[0221] Figure 17 An example of a computing device 1700 according to various aspects of this disclosure is shown. The example shown includes a computing device 1700, a processor 1705, a memory subsystem 1710, a communication interface 1715, an I / O interface 1720, a user interface component 1725, and a channel 1730.

[0222] In some embodiments, computing device 1700 is a reference Figure 1 and Figure 16 Examples of the described image processing apparatus, or including references Figure 1 and Figure 16 The image processing apparatus described herein. In some embodiments, computing device 1700 includes processor 1705, which is capable of executing instructions stored in memory subsystem 1710 to obtain an input image and modification prompts, generate text embeddings based on the modification prompts, and generate modified images based on the input image and text embeddings.

[0223] According to some embodiments, processor 1705 includes one or more processors. In some cases, processor 1705 is a smart hardware device (e.g., a general-purpose processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or a combination thereof). In some cases, processor 1705 is configured to use a memory controller to operate a memory array. In other cases, the memory controller is integrated into processor 1705. In some cases, processor 1705 is configured to execute computer-readable instructions stored in memory to perform various functions. In some embodiments, processor 1705 includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing. Processor 1705 is a reference. Figure 16 Examples of the described processor units, or including references Figure 16 The processor unit described.

[0224] According to some embodiments, memory subsystem 1710 includes one or more memory devices. Examples of memory devices include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of memory devices include solid-state memory and hard disk drives. In some examples, the memory is used to store computer-readable, computer-executable software including instructions that, when executed, cause a processor to perform the various functions described herein. In some cases, among other things, the memory includes a basic input / output system (BIOS) that controls basic hardware or software operations, such as interaction with peripheral components or devices. In some cases, a memory controller operates memory cells. For example, a memory controller can include row decoders, column decoders, or both. In some cases, memory cells within the memory store information in the form of logical states. Memory subsystem 1710 is a reference. Figure 16Examples of the described memory cells, or including references Figure 16 The described aspects of the memory cell.

[0225] According to some embodiments, communication interface 1715 operates at the boundary between a communication entity (such as computing device 1700, one or more user devices, a cloud, and one or more databases) and channel 1730, and is capable of recording and processing communications. In some cases, communication interface 1715 is provided to allow a processing system to be coupled to a transceiver (e.g., a transmitter and / or receiver). In some examples, the transceiver is configured to transmit (or send) and receive signals for the communication device via an antenna. In some cases, a bus is used in communication interface 1715.

[0226] According to some embodiments, I / O interface 1720 is controlled by an I / O controller to manage input and output signals for computing device 1700. In some cases, I / O interface 1720 manages peripheral devices not integrated into computing device 1700. In some cases, I / O interface 1720 represents a physical connection or port to an external peripheral device. In some cases, the I / O controller uses an operating system (such as iOS®, ANDROID®, MS-DOS®, MS-WINDOWS®, OS / 2®, UNIX®, LINUX®, or other known operating systems). In some cases, the I / O controller represents, or interacts with, a modem, keyboard, mouse, touchscreen, or similar device. In some cases, the I / O controller is implemented as a component of the processor. In some cases, a user interacts with the device via I / O interface 1720 or hardware components controlled by the I / O controller. I / O interface 1720 is a reference. Figure 16 Examples of the described I / O modules, or including references Figure 16 The various aspects of the described I / O module.

[0227] According to some embodiments, user interface component 1725 enables a user to interact with computing device 1700. In some cases, user interface component 1725 includes audio devices (such as external speaker systems), external display devices (such as displays), input devices (e.g., remote control devices that interface with the user interface directly or via an I / O controller), or combinations thereof.

[0228] The performance of the apparatus, system, and method of this disclosure has been evaluated, and the results indicate that embodiments of this disclosure have achieved improved performance compared to conventional techniques (e.g., conventional image generation models). Example experiments demonstrate that the image processing apparatus based on this disclosure outperforms conventional image generation models. References Figures 4 to 5Details of example use cases based on embodiments of this disclosure are described.

[0229] The descriptions and accompanying drawings described herein represent exemplary configurations and do not represent all implementations within the scope of the claims. For example, operations and steps may be rearranged, combined, or otherwise modified. Furthermore, structures and devices may be represented in block diagram form to illustrate relationships between components and avoid obscuring the described concepts. Similar components or features may have the same name but may have different reference numerals corresponding to different drawings.

[0230] Some modifications to this disclosure will be apparent to those skilled in the art, and the principles defined herein can be applied to other variations without departing from the scope of this disclosure. Therefore, this disclosure is not limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0231] The described methods can be implemented or performed by devices including general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor can be a microprocessor, a conventional processor, a controller, a microcontroller, or a state machine. A processor can also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration). Therefore, the functions described herein can be implemented in hardware or software and can be performed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions can be stored on a computer-readable medium in the form of instructions or code.

[0232] Computer-readable media include both non-transitory computer storage media and communication media, with communication media encompassing any medium that facilitates the transfer of code or data. Non-transitory storage media can be any available medium accessible by a computer. For example, non-transitory computer-readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CDs or other optical storage devices, magnetic disk storage devices, or any other non-transitory medium used to carry or store data or code.

[0233] Furthermore, connecting components can be appropriately referred to as computer-readable media. For example, if code or data is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared, radio, or microwave signals), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology is included in the definition of media. Combinations of media are also included within the scope of computer-readable media.

[0234] In this disclosure and the following claims, the word "or" indicates a list of inclusion properties, such that a list of, for example, X, Y, or Z means X or Y or Z or XY or XZ or YZ or XYZ. The phrase "based on" is also not used to indicate a closed set of conditions. For example, a step described as "based on condition A" could be based on both condition A and condition B. In other words, the phrase "based on" should be interpreted as meaning "at least partially based on". Furthermore, the words "a" or "an" indicate "at least one".

Claims

1. A method comprising: Obtain an input image and a modification prompt, wherein the input image depicts an object having a first attribute, and the modification prompt describes a modification from the first attribute to a second attribute different from the first attribute; A text encoder is used to encode the modified prompt word to obtain a text embedding, wherein the text embedding represents the modification in the embedding space; as well as Based on the input image and the text embedding, an image generation model is used to generate a modified image, wherein the modified image depicts the object having the second attribute.

2. The method according to claim 1, wherein: The modified image retains the third attribute of the object.

3. The method according to claim 1, further comprising: Obtain a mask input indicating a region of the input image, wherein the modified image is generated based on the mask input.

4. The method according to claim 1, further comprising: Obtain reference input; as well as A reference embedding is generated based on the reference input, wherein the modified image is generated based on the reference embedding.

5. The method according to claim 1, further comprising: Obtain noise input; as well as The noise input is denoised based on the text embedding to obtain the modified image.

6. The method according to claim 1, further comprising: The color of the input image is modified to obtain a modified input image, wherein the modified image is generated based on the modified input image.

7. The method according to claim 1, further comprising: Generate a first image feature based on the input image; A second image feature is generated based on the text embedding; as well as The first image feature is combined with the second image feature to obtain a combined image feature, wherein the modified image is generated based on the combined image feature.

8. The method according to claim 1, wherein: The image generation model is trained to edit images using a training set, wherein the training set includes training images depicting the object having the first attribute and training cue words describing the modifications from the first attribute to the second attribute.

9. The method according to claim 1, wherein: The image generation model is trained using a first predicted image generated based on the captions and a second predicted image generated based on changes to the captions.

10. A method for training a machine learning model, the method comprising: Obtain a training set, which includes training images, modification cue words describing changes to the training images, and modified training images depicting the changes to the training images. A text encoder is used to encode the modified prompt word to obtain a text embedding, wherein the text embedding represents the modification in the embedding space; as well as The training set and the text embedding are used to train an image generation model to generate synthetic images based on the modified prompts.

11. The method of claim 10, wherein: The image generation model is trained to retain elements in the training images that are not described by the modified prompt words.

12. The method according to claim 10, wherein: The training set includes mask inputs indicating regions of the training images, wherein the image generation model is trained to generate the synthetic images based on the mask inputs.

13. The method of claim 10, wherein training the image generation model comprises: A first predicted image is generated based on the captions for the training images; as well as A second predicted image is generated based on the modified prompt words.

14. The method of claim 10, further comprising: Generate first image features based on the training images; Generate second image features based on the modified prompt words; as well as The first image feature and the second image feature are combined to obtain a combined image feature.

15. The method of claim 10, wherein training the image generation model comprises: Calculate the diffusion loss, wherein the image generation model is trained based on the diffusion loss.

16. The method of claim 10, wherein obtaining the training set comprises: The color of the training image is modified to obtain the modified training image.

17. A system comprising: Memory components; as well as A processing device coupled to the memory component, the processing device being configured to perform operations including: Obtain an input image and a modification prompt, wherein the input image depicts an object having a first attribute, and the modification prompt describes a modification from the first attribute to a second attribute different from the first attribute; A text encoder is used to encode the modified cue words to obtain a text embedding, wherein the text embedding represents the modification in the embedding space; and Based on the input image and the text embedding, an image generation model is used to generate a modified image, wherein the modified image depicts the object having the second attribute.

18. The system according to claim 17, wherein: The image generation model includes a diffusion model.

19. The system according to claim 17, wherein: The modified image retains the third attribute of the object.

20. The system of claim 17, further comprising: A reference encoder configured to generate a reference embedding based on a reference input, wherein the modified image is generated based on the reference embedding.

Images