Reference image structure matching using diffusion model

Abstract

The embodiment of the invention relates to reference image structure matching using a diffusion model. The invention discloses a method, a device, a non-transitory computer readable medium and a system for image processing. The method comprises the following steps: acquiring a structure input indicating a target space structure; encoding the structural input using a conditional encoder to obtain a structural code representing the target spatial structure; and generating a composite image based on the structure encoding using the image generation model, wherein the composite image depicts the object having the target spatial structure.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 569,890 filed in the U.S. Patent and Trademark Office on March 26, 2024, and Regular Patent Application No. 18 / 947,959 filed on November 14, 2024, the disclosures of which are incorporated herein by reference in their entirety. Background Art

[0003] The following is generally related to image processing, and more specifically, image generation using machine learning models. Image processing refers to the use of computers to edit images using algorithms or processing networks. In some cases, image processing software can be used for a variety of image processing tasks, such as image restoration, image detection, image editing, image synthesis, and image generation. For example, image generation involves using machine learning models to generate synthetic images based on adjustments.

[0004] In the field of image generation, an input image and a text prompt are provided to a machine learning model to generate a synthetic image that includes the image structure described in the input image. In some cases, the input image includes image elements that depict the structural input. In some cases, the text prompt describes the image elements to be generated that have the structural input. However, in some cases, extensive training of the image generation model may be required to perform the structural matching task. Summary of the Invention

[0005] Aspects of the present disclosure provide a method and system for image generation. In one aspect, the system receives an input image describing a spatial structure and a textual prompt describing an image element, and generates a composite image describing the image element having the spatial structure. According to some aspects, the system includes a conditional encoder configured to generate a structural encoding representing the input spatial structure. In some aspects, the system includes an image generation model that is trained to generate output features by combining the structural encoding and intermediate features generated in each encoding layer of a U-net of the image generation model. In some cases, the system decodes the output features to generate a composite image. By combining the structural encoding and the intermediate features at each encoding layer, the image generation model ensures that the composite image accurately depicts the target spatial structure from the input image and the image elements described by the textual prompt.

[0006] A method, apparatus, non-transitory computer-readable medium, and system for image processing, comprising: obtaining a structure input indicating a target spatial structure; encoding the structure input using a conditional encoder to obtain a structure encoding representing the target spatial structure; and generating a synthetic image based on the structure encoding using an image generation model, wherein the synthetic image depicts an object having the target spatial structure.

[0007] A method, apparatus, non-transitory computer-readable medium, and system for training a machine learning model, comprising: obtaining a training set comprising a training structure input indicating a spatial structure and a reference truth image comprising the spatial structure; and using the training set to train an image generation model to generate a synthetic image based on the structure input, wherein the synthetic image comprises the spatial structure.

[0008] An apparatus and system for image processing include a memory component and a processing device coupled to the memory component, the processing device being configured to perform operations including: obtaining a structure input indicating a target spatial structure, encoding the structure input using a conditional encoder to obtain a structure encoding representing the target spatial structure, and generating a synthetic image based on the structure encoding using an image generation model, wherein the synthetic image depicts an object having the target spatial structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An example of an image processing system according to various aspects of the present disclosure is shown.

[0010] Figure 2 An example of a method of generating a composite image according to aspects of the present disclosure is shown.

[0011] Figure 3 An example of image generation based on depth adjustment according to aspects of the present disclosure is shown.

[0012] Figure 4 An example of image generation based on edge adjustment according to aspects of the present disclosure is shown.

[0013] Figure 5 Examples of image generation based on different levels of structural adhesion according to aspects of the present disclosure are shown.

[0014] Figure 6 An example of a method for performing image processing according to aspects of the present disclosure is shown.

[0015] Figure 7 An example of an image processing apparatus according to aspects of the present disclosure is shown.

[0016] Figure 8 Examples of machine learning models according to aspects of the present disclosure are shown.

[0017] Figure 9 An example of image generation based on a reference image according to aspects of the present disclosure is shown.

[0018] Figure 10 An example of image generation based on entity control according to aspects of the present disclosure is shown.

[0019] Figure 11An example of image generation based on a user sketch according to aspects of the present disclosure is shown.

[0020] Figure 12 Examples of image generation models according to aspects of the present disclosure are shown.

[0021] Figure 13 A U-Net example according to aspects of the present disclosure is shown.

[0022] Figure 14 An example diffusion process according to aspects of the present disclosure is shown.

[0023] Figure 15 An example of a method of generating a synthetic image using a structure encoder according to aspects of the present disclosure is shown.

[0024] Figure 16 Examples of methods for training machine learning models according to aspects of the present disclosure are shown.

[0025] Figure 17 An example of a graphical depiction according to aspects of the present disclosure is shown.

[0026] Figure 18 An example of a flowchart depicting an algorithm as a step-by-step process in an example implementation of operations that may be performed for training a machine learning model in accordance with aspects of the present disclosure is shown.

[0027] Figure 19 An example of a method for training a diffusion model according to aspects of the present disclosure is shown.

[0028] Figure 20 An example of a computing device according to aspects of the present disclosure is shown. DETAILED DESCRIPTION

[0029] Aspects of the present disclosure relate to image generation using generative machine learning. Embodiments of the present disclosure relate to image generation systems that accurately generate images that depict objects described by textual prompts and have spatial structure from input images. In certain aspects, the system includes a conditional encoder trained to generate a structure encoding based on the input image; and an image generation model trained to generate a synthetic image based on the structure encoding. The structure encoding generated by the conditional encoder is provided to an encoding layer of the image generation model to ensure that the target structure encoding from the input image is maintained in or transferred to the synthetic image.

[0030] Some traditional image generation models, such as the ControlNet, include multiple convolutional layers, self-attention layers, and cross-attention layers. Due to the complex architecture, traditional methods cannot train the entire image generation model (including the U-Net, convolutional layers, self-attention layers, and cross-attention layers). In some cases, the U-Net may not be trained. As a result, traditional image generation models require a long processing time to generate output images. In some cases, due to lack of training, the output image may not describe the conditional input.

[0031] Therefore, embodiments of the present disclosure provide a system and method that improves conventional image generation systems by accurately generating synthetic images from input images, where the synthetic images depict objects described by textual cues and have spatial structure. This is achieved using a system that includes a conditional encoder trained to generate structured encodings and an image generation model trained to generate synthetic images based on the structured encodings.

[0032] According to an embodiment of the present disclosure, a machine learning system receives an input image and a text prompt to generate a composite image. For example, the system includes a feature extractor configured to extract a feature map from the input image. The feature map is used as a structural input to an image generation model to adjust the image generation process. For example, a feature encoder encodes the structural input to obtain a structural encoding. The structural encoding is combined with a noise input (e.g., a noise input to the image generation model) to obtain a combined structural encoding.

[0033] In some embodiments, the combined structural encoding is input to a conditional encoder to generate layer-specific conditional encoding (e.g., structural encoding). For example, the layer-specific conditional encoding is combined with the corresponding downsampling layer of the U-Net architecture of the image generation model. In one aspect, the layer of the conditional encoder includes a convolutional layer or an activation layer. After performing multiple downsampling processes, the combined encoding is upsampled via an upsampling layer to generate output features. In one aspect, the image generation model generates a synthetic image based on the output features. In one aspect, the synthetic image includes a spatial structure indicated by a structural input from an input image and an element described by a text prompt.

[0034] refer to Figure 1 and Figure 20 An example system of the present disclosure in image processing is provided. Figures 2 to 5 Provides an example application of the present disclosure in image processing. Figures 7 to 13 Provides details about the architecture of the image processing device. Figure 6 and Figures 14 and 15 Provides examples for image processing procedures. Figures 16 to 19 Provides a description of the example training process.

[0035] Therefore, the present disclosure provides a system and method that improves traditional image editing systems by more accurately and efficiently generating synthetic images that depict the target spatial structure. For example, the conditional encoder includes a convolutional layer or an activation layer (instead of the additional self-attention layer and cross-attention layer included in the traditional system). Because the conditional encoder has one or more parameters with fewer magnitudes than the conventional system, the processing time for generating the synthetic image is reduced. During training, the conditional encoder and the image generation model (e.g., U-Net) are jointly trained without increasing the training cost due to the fewer model parameters of the conditional encoder. As a result, the system is able to accurately and efficiently generate synthetic images that describe the target spatial structure.

[0036] Image processing

[0037] exist Figures 1 to 6 and 14 to Figure 15 In the present invention, a method, apparatus, non-transitory computer-readable medium, and system for image processing include: obtaining a structure input indicating a target spatial structure; encoding the structure input using a conditional encoder to obtain a structure encoding representing the target spatial structure; and generating a synthetic image based on the structure encoding using an image generation model, wherein the synthetic image depicts an object having the target spatial structure.

[0038] Some examples of the methods, apparatuses, non-transitory computer-readable media, and systems further include encoding each of a plurality of components of the structural input to obtain a plurality of component structural encodings, wherein each of the plurality of components comprises a different representation of the target spatial structure. Some examples further include combining the plurality of component structural encodings to obtain a preliminary structural encoding. Some examples further include encoding the preliminary structural encoding using a conditional encoder to obtain the structural encoding.

[0039] In some aspects, each component structural encoding in the plurality of component structural encodings is generated by a different structural encoder. In some aspects, each component structural encoding in the plurality of component structural encodings has a different number of channels. In some aspects, the plurality of component structural encodings includes depth encoding, edge encoding, and entity encoding.

[0040] Some examples of the methods, apparatuses, non-transitory computer-readable media, and systems further include providing the structure encoding to a first layer of the image generation model. Some examples further include downsampling the structure encoding to obtain a downsampled structure encoding. Some examples further include providing the downsampled structure encoding to a second layer of the image generation model.

[0041] Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include obtaining a noisy input. Some examples further include denoising the noisy input based on the structured code. Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include performing multiple convolutions on the structured code.

[0042] Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include obtaining structural adhesion parameters, wherein the composite image is generated based on the structural adhesion parameters using structural encoding. Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include obtaining a textual cue describing the object, wherein the composite image is generated based on the textual cue.

[0043] Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include obtaining a style hint indicating a style element, wherein generating a composite image based on the style hint includes the style element. Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include obtaining a preliminary image. Some examples further include generating a structure input based on the preliminary image.

[0044] According to some aspects, a method, apparatus, non-transitory computer-readable medium, and system for image processing are described. One or more aspects of the method, apparatus, non-transitory computer-readable medium, and system include: obtaining a structure input indicating spatial structure; encoding the structure input using a conditional encoder to obtain a structure encoding; and generating a synthetic image based on the structure encoding using an image generation model, wherein the synthetic image depicts an object having the spatial structure.

[0045] Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include obtaining a plurality of structural inputs. Some examples further include encoding each of the plurality of structural inputs to obtain a plurality of structural codes, wherein the composite image is generated based on the plurality of structural codes.

[0046] In some aspects, each of the plurality of structural codes is generated by a different structural encoder. In some aspects, each of the plurality of structural codes has a different number of channels. In some aspects, the plurality of structural codes includes a depth code, an edge code, and an entity code.

[0047] Some examples of the methods, apparatuses, non-transitory computer-readable media, and systems further include providing the structure encoding to a first layer of the image generation model. Some examples further include downsampling the structure encoding to obtain a downsampled structure encoding. Some examples further include providing the downsampled structure encoding to a second layer of the image generation model.

[0048] Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include combining the structured code with a noise input to obtain a modified noise input, wherein a composite image is generated based on the modified noise input. Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include performing multiple convolutions on the structured code.

[0049] Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include obtaining structural adhesion parameters, wherein the composite image is generated based on the structural adhesion parameters using structural encoding. Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include obtaining textual cues, wherein the composite image is generated based on the textual cues. Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include obtaining style cues, wherein the composite image is generated based on the style cues.

[0050] Figure 1 An example of an image processing system according to aspects of the present invention is shown. The example shown includes a user 100, a user device 105, an image processing apparatus 110, a cloud 115, and a database 120. The image processing apparatus 110 is a reference Figure 7 Examples of corresponding elements are described or include aspects thereof.

[0051] Reference Figure 1 , the user 100 provides an input image and a text prompt to the image processing device 110 via the user device 105 and the cloud 115. In some cases, the text prompt describes the elements to be depicted in the synthetic image to be generated. In some embodiments, the machine learning model extracts a feature map from the input image as a condition for the image generation model. In some cases, the feature map indicates a spatial structure. For example, the feature map includes a depth map, an edge map, a scribble map, an entity map, or a combination thereof. The feature map is fed to the conditional encoder and combined with the intermediate features of the U-Net at each downsampling layer of the encoding layer of the U-Net. The image processing device 110 generates a synthetic image based on the spatial structure from the input image and depicts the image elements described by the text prompt. The synthetic image is displayed to the user 100 via the user device 105 and the cloud 115.

[0052] User device 105 may be a personal computer, laptop computer, mainframe computer, palmtop computer, personal assistant, mobile device, or any other suitable processing device. In some examples, user device 105 includes software incorporating an image processing application. In some examples, the image processing application on user device 105 may include the functionality of image processing apparatus 110.

[0053] The user interface can enable user 100 to interact with user device 105. In some embodiments, the user interface can include an audio device such as an external speaker system, an external display device such as a display screen, or an input device (e.g., a remote control device that interfaces with the user interface directly or through an I / O controller module). In some cases, the user interface can be a graphical user interface (GUI). In some examples, the user interface can be represented by code, where the code is sent to user device 105 and drawn locally by the browser. Figure 2 The processing using the image processing device 110 is also described. The user interface is referenced Figures 3 to 5 Examples of image generation systems are described, or include aspects thereof.

[0054] According to some aspects, the image processing device 110 includes a computer-implemented network that includes a machine learning model, a conditional encoder, and an image generation model. The image processing device 110 also includes a processor unit, a memory unit, an I / O module, and a training component. In some embodiments, the image processing device 110 also includes a communication interface, a user interface component, and a bus, as described in reference to FIG. Figure 17 Additionally, the image processing apparatus 110 communicates with the user device 105 and the database 120 via the cloud 115. The image processing apparatus 110 is a reference Figure 7 Examples of corresponding elements described, or including references Figure 7 Describes aspects of the corresponding elements. Figure 2 More details are provided regarding the operation of the image processing apparatus 110 .

[0055] In some cases, the image processing device 110 is implemented on a server. The server provides one or more functions to the user through one or more network links in various networks. In some cases, the server includes a single microprocessor board, which includes a microprocessor responsible for controlling various aspects of the server. In some cases, the server uses a microprocessor and protocol to exchange data with other devices / users on one or more networks through Hypertext Transfer Protocol (HTTP) and Simple Mail Transfer Protocol (SMTP), but other protocols such as File Transfer Protocol (FTP) and Simple Network Management Protocol (SNMP) can also be used. In some cases, the server is configured to send and receive files (e.g., for displaying web pages) in Hypertext Markup Language (HTML) format. In various embodiments, the server includes a general-purpose computing device, a personal computer, a laptop computer, a mainframe computer, a supercomputer, or any other suitable processing device.

[0056] Cloud 115 is a computer network configured to provide on-demand availability of computer system resources such as data storage and computing power. In some examples, cloud 115 provides resources that do not require active management by users (e.g., user 100). The term cloud is sometimes used to describe a data center that is available to many users via the Internet. Some large cloud networks have functions distributed across multiple locations from a central server. A server is designated as an edge server if it has a direct or close connection to a user. In some cases, cloud 115 is limited to a single organization. In other examples, cloud 115 can be used for many organizations. In one example, cloud 115 includes a multi-layer communication network that includes multiple edge routers and core routers. In another example, cloud 115 is based on a local collection of switches in a single physical location.

[0057] According to some aspects, database 120 stores training data (or training sets) including training structure inputs and reference truth images. Database 120 is an organized collection of data. For example, database 120 stores data in a specified format called a schema. Database 120 can be constructed as a single database, a distributed database, multiple distributed databases, or an emergency backup database. In some cases, a database controller can manage the storage and processing of data in database 120. In some cases, a user (e.g., user 100) interacts with the database controller. In other cases, the database controller can operate automatically without user interaction.

[0058] Figure 2 An example of a method 200 for generating a composite image according to aspects of the present disclosure is shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of the device. Additionally or alternatively, dedicated hardware is used to perform certain processes. Generally, these operations are performed in accordance with the methods and processes described in various aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in conjunction with other operations.

[0059] At operation 205, the system provides an input image and text prompt. In some cases, the operation of this step refers to or can be referred to Figure 1 The user may perform a description of the image processing. For example, the input image depicts a black and white line drawing of a bird. For example, the text prompt states "Photograph of a red bird sitting on a branch surrounded by lush green leaves." In some cases, a style prompt indicating the synthesized image may be provided to the image processing device. In some cases, additional keywords such as "photograph" or "photorealistic" may be provided to the image processing device. In some cases, other parameters such as an aesthetic score and / or text weight may be provided to the image processing device.

[0060] At operation 210, the system generates the adjustment guidance code. In some cases, the operation of this step refers to or can be referred to by Figure 1 and Figure 7 In some cases, the operation of this step refers to or can be performed by referring to Figure 7 and Figure 8 In some cases, for example, the system may extract one or more feature maps (e.g., a structure input) based on the input image. In some cases, the feature maps include a depth map, an edge map, and / or a solid map. In some cases, the system includes a feature encoder that generates a structure encoding based on the structure input, respectively. In some embodiments, the system combines the structure encodings to obtain a combined structure encoding, wherein the combined structure encoding is used to guide the image generation process. Figure 8 Describes more details about structure encoding.

[0061] At operation 215, the system initializes the noise input. In some cases, the operation of this step refers to or can be referred to Figure 1 and 7 In some cases, the operation of this step refers to or can be performed by referring to Figure 7 、 Figures 9 to 11 and Figure 17 In some cases, the image generation model is initialized to include a noise input of random noise. The noise input can be in a latent space. By initializing the image generation model with random noise, different variations of synthetic images including content described by textual conditioning (e.g., textual prompts) can be generated. In some cases, a text encoding or text embedding of the textual prompt is combined with the noise features using a cross-attention block within the image generation model to guide the image generation process. Figure 12 Describes more details about the image generation process.

[0062] At operation 220, the system generates media content. In some cases, the operation of this step refers to or can be referred to by Figure 1 and Figure 7 In some cases, the operation of this step refers to or can be performed by referring to Figure 7 、 Figures 9 to 11 and Figure 17 The image generation model described herein is used to perform the processing. For example, the media content includes a composite image or a modified image, each depicting a red bird surrounded by green leaves. For example, the composite image includes image pixels generated by the image generation model. For example, the modified image includes image pixels from the input image and image pixels generated by the image generation model. In some cases, the media content is displayed to a user via a user device.

[0063] Figure 3 An example of depth-modulated image generation according to aspects of the present disclosure is shown. The example shown includes an image generation system 300, a text prompt 305, an input image 310, a depth map 315, a synthesized image 320, and a machine learning model 325. In some cases, the image generation system 300 can be used in reference to Figure 1 The described user interface or user device is implemented.

[0064] Reference Figure 3 , a text prompt 305 and an input image 310 are provided to an image generation system 300 to generate a composite image 320. For example, the text prompt 305 states "cartoon of a tiger" and the input image 310 depicts a photo of a tiger. In some cases, the machine learning model 325 includes a depth model configured to extract a depth map 315 from the input image 310, wherein the depth map 315 is used as a structural input to adjust the image generation process. Figure 3 As shown, the composite image 320 has the same spatial structure as the input image 310. In some cases, the composite image 320 depicts one or more elements described by the text prompt 305. In some cases, the composite image 320 includes an image variation of a tiger. Figure 8 Describes the image generation process in more detail.

[0065] The image generation system 300 is a reference Figure 4 and Figure 5 Examples of corresponding elements described herein, or including aspects thereof. Text hint 305 is a reference to Figure 4 、 Figure 5 、 Figures 9 to 11 and Figure 17 The input image 310 is a reference image. Figure 4 、 Figure 5 、 Figure 9 and Figure 10 Examples of corresponding elements are described or include aspects thereof.

[0066] The depth map 315 is a reference Figure 5 、 Figure 9 and Figure 17 The composite image 320 is a reference to an example of a corresponding element described in the embodiment of the present invention, or includes aspects thereof. Figure 4 and Figure 5 Examples of corresponding elements described herein or including aspects thereof. Machine learning model 325 is a reference Figure 4 、 Figure 5 、 Figure 7 and Figure 8 Examples of corresponding elements are described or include aspects thereof.

[0067] Figure 4 An example of edge-condition-based image generation according to aspects of the present disclosure is shown. The example shown includes an image generation system 400, a text prompt 405, an input image 410, an edge map 415, a synthesized image 420, and a machine learning model 425. In some cases, the image generation system 400 can be used in reference to Figure 1 The described user interface or user device is implemented.

[0068] Reference Figure 4 , a text prompt 405 and an input image 410 are provided to an image generation system 400 to generate a composite image 420. For example, the text prompt 405 indicates "outdoor photo of an old house," and the input image 410 depicts a cake. In some cases, the machine learning model 425 includes an edge model configured to extract an edge map 415 from the input image 410, wherein the edge map 415 is used as a structural input to adjust the image generation process. Figure 4 As shown, the composite image 420 has the same spatial structure as the input image 410. In some cases, the composite image 420 depicts one or more elements described by the text prompt 405. In some cases, the composite image 420 includes an image variation of the old house. More details about the image generation process are referred to Figure 8 describe.

[0069] Image generation system 400 is a reference Figure 3 and Figure 5 Examples of corresponding elements described herein, or including aspects thereof. Text hint 405 is a reference to Figure 3 、 Figure 5 、 Figures 9 to 11 and Figure 17 The input image 410 is a reference image. Figure 3 、 Figure 5 、 Figure 9 and Figure 10 Examples of corresponding elements are described or include aspects thereof.

[0070] Edge map 415 is a reference Figure 5 、 Figure 9 and Figure 17 The composite image 420 is a reference to an example of a corresponding element or includes aspects thereof. Figure 3 and Figure 5 Examples of corresponding elements described herein or including aspects thereof. Machine learning model 425 is a reference Figure 3 、 Figure 5 、 Figure 7 and Figure 8 Examples of corresponding elements are described or include aspects thereof.

[0071] Figure 5An example of image generation based on various levels of structural adhesion according to aspects of the present disclosure is shown. The example shown includes an image generation system 500, a text prompt 505, an input image 510, an edge map 515, a depth map 520, a synthesized image 525, and a machine learning model 530. In some cases, the image generation system 500 can be used in reference to Figure 1 The described user interface or user device is implemented.

[0072] Reference Figure 5 , a text prompt 505 and an input image 510 are provided to an image generation system 500 to generate a composite image 525. For example, the text prompt 505 states "architectural photography of uniquely shaped buildings under the moon," and the input image 510 depicts a dog. In some cases, the machine learning model 530 includes an edge model and a depth model, each configured to extract an edge map 515 and a depth map 520, respectively, from the input image 510, wherein the edge map 515 and the depth map 520 are used as structural inputs to adjust the image generation process. Figure 5 As shown, the composite image 525 has the same spatial structure as the input image 510. In some cases, the composite image 525 depicts one or more elements described by the text prompt 505. In some cases, the composite image 525 includes an image variant of a uniquely shaped building. More details about the image generation process are referred to Figure 8 describe.

[0073] Image generation system 500 is a reference Figure 3 and Figure 4 Examples of corresponding elements described herein or including aspects thereof. Text hint 505 is a reference to Figure 3 、 Figure 4 、 Figures 9 to 11 and Figure 17 The input image 510 is a reference image. Figure 3 、 Figure 4 、 Figure 9 and Figure 10 Examples of corresponding elements are described or include aspects thereof.

[0074] Edge map 515 is a reference Figure 4 、 Figure 9 and Figure 17 Depth map 520 is a reference to the corresponding element of the example or includes aspects thereof. Figure 3 、 Figure 9 and Figure 17 Examples of corresponding elements described herein or including aspects thereof. Composite image 525 is a reference Figure 3 and Figure 4 Examples of corresponding elements described herein or including aspects thereof. The machine learning model 530 is a reference Figure 3 、 Figure 4 、 Figure 7 and Figure 8 Examples of corresponding elements are described or include aspects thereof.

[0075] Figure 6 An example of a method 600 for image processing according to various aspects of the present disclosure is shown. In some examples, these operations are performed by a system comprising a processor that executes a set of code to control the functional elements of a device. Additionally or alternatively, dedicated hardware is used to perform certain processes. Typically, these operations are performed according to the methods and processes described in various aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed together with other operations.

[0076] At operation 605, the system obtains a structure input indicating a target spatial structure. In some cases, the operation of this step involves or can be performed by reference to Figures 3 to 5 、 Figure 7 and Figure 8 The machine learning model described is performed. In some cases, the structural input includes a depth map, an edge map, and an entity map. For example, a depth map is a grayscale image in which each pixel encodes the distance or depth of the corresponding point in the scene. The depth map represents the spatial layout of objects in the scene based on the distance from the viewpoint. For example, an edge map is a binary or grayscale image that represents the boundaries or edges of objects within the scene. Each pixel in the edge map represents the presence or strength of an edge in the corresponding position of the input image. For example, the entity map highlights entities (e.g., objects, people, background scenes, etc.) within the input image. In some cases, the entity map is represented as an RGB color map.

[0077] In some cases, spatial structure or target spatial structure refers to the arrangement and layout of visual elements within an image. For example, spatial structure includes features such as spatial patterns or spatial relationships of objects, regions, or features. In some cases, spatial structure can be encoded in depth maps, edge maps, and entity maps.

[0078] In some cases, for example, an image element is an image component or image feature that constitutes the overall composition of the image, such as an object, entity, subject, shape, color, texture, pattern, background scene, visual attributes and / or style. For example, an image element can be an animal, such as a cat or dog, a person, an object, such as a hat or a table, a scene, such as a beach or a mountaintop, or a combination thereof. For example, in some cases, an image element can indicate the configuration, style, color scheme, lighting effect, perspective, angle of view, texture, or composition rules of the image.

[0079] At operation 610, the system encodes the structure input using a conditional encoder to obtain a structure encoding representing the target spatial structure. In some cases, the operation of this step involves or can be performed by reference Figure 7 and Figure 8 In some cases, conditioning is performed by converting (encoding or embedding) the input into an encoding so that the machine learning model can more easily process the data. In some cases, the encoding comprises a vector representation of the input. In some cases, the encoding is in a digital format that allows the machine learning model to process the data more effectively and efficiently. In some cases, the encoding can be a latent representation or latent code. In some cases, the encoding can include a feature map. In some cases, conditioning refers to the process of incorporating additional information or constraints into the machine learning model so that the output (usually a synthesized image) closely follows the additional information.

[0080] At operation 615, the system generates a synthetic image based on the structure encoding using the image generation model, wherein the synthetic image depicts an object having the target spatial structure. In some cases, the operation of this step involves or can be performed by reference Figure 7 、 Figures 9 to 11 and Figure 17 The system may be configured to generate a modified image based on the structured encoding. For example, the composite image may include image pixels generated by the image generation model. For example, the modified image may include image pixels from the input image and image pixels generated by the image generation model.

[0081] System Architecture

[0082] exist Figures 7 to 13 and Figure 20 In the present invention, an apparatus and system for image processing includes a memory component and a processing device coupled to the memory component, wherein the processing device is configured to perform operations including: obtaining a structure input indicating a target spatial structure, encoding the structure input using a conditional encoder to obtain a structure encoding representing the target spatial structure, and generating a synthetic image based on the structure encoding using an image generation model, wherein the synthetic image depicts an object having the target spatial structure.

[0083] In some aspects, the conditional encoder includes a plurality of convolutional layers and a plurality of activation layers. In some aspects, the image generation model includes more parameters than the conditional encoder. Some examples of the apparatus and system also include a text encoder configured to encode the text prompt to obtain a text embedding.

[0084] According to certain aspects, an apparatus and system for image processing are described. One or more aspects of the apparatus and system include at least one processor; at least one memory storing instructions executable by the at least one processor; a conditional encoder including parameters stored in the at least one memory and trained to encode a structure input to obtain a structure encoding, wherein the structure input indicates spatial structure; and an image generation model including parameters stored in the at least one memory and trained to generate a synthetic image based on the structure encoding, wherein the synthetic image depicts an object having spatial structure.

[0085] In some aspects, each layer of the conditional encoder comprises a convolutional layer or an activation layer. In some aspects, the image generation model comprises at least ten times more parameters than the conditional encoder. Some examples of the apparatus and system also include a text encoder configured to encode the text prompt to obtain a text embedding.

[0086] Figure 7 An example of an image processing apparatus 700 according to aspects of the present disclosure is shown. The shown example includes the image processing apparatus 700, a processor unit 705, an I / O module 710, a memory unit 715, and a training component 735.

[0087] According to some embodiments of the present disclosure, the image processing device 700 includes a computer-implemented artificial neural network (ANN). An ANN is a hardware or software component that includes a plurality of connected nodes (e.g., artificial neurons) that loosely correspond to neurons in the human brain. Each connection or edge sends a signal from one node to another (such as a physical synapse in the brain). When a node receives a signal, the node processes the signal and then sends the processed signal to other connected nodes. In some cases, the signals between nodes include real numbers, and the output of each node is calculated as a function of the sum of its inputs. In some examples, the nodes may determine the output using other mathematical algorithms (e.g., selecting the maximum value from the input as the output) or any other suitable algorithm for activating the node. Each node and edge is associated with one or more node weights that determine how to process and send the signal. The image processing device 700 is a reference Figure 1 Examples of corresponding elements described, or including references Figure 1 Describes aspects of the corresponding elements.

[0088] The processor unit 705 is an intelligent hardware device (e.g., a general-purpose processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof). In some cases, the processor unit 705 is configured to operate a memory array using a memory controller. In other cases, the memory controller is integrated into the processor. In some cases, the processor unit 705 is configured to execute computer-readable instructions stored in the memory to perform various functions. In some embodiments, the processor unit 705 includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing. The processor unit 705 is a reference Figure 20 Examples of processors are described or include aspects thereof.

[0089] The I / O module 710 (e.g., input / output interface) may include an I / O controller. The I / O controller may manage input and output signals for the device. The I / O controller may also manage peripheral devices that are not integrated into the device. In some cases, the I / O controller may represent a physical connection or port to an external peripheral device. In some cases, the I / O controller may use a device such as or other known operating systems. In other cases, an I / O controller may represent or interact with a modem, keyboard, mouse, touch screen, or similar device. In some cases, an I / O controller may be implemented as part of a processor. In some cases, a user may interact with a device through the I / O controller or via hardware components controlled by the I / O controller.

[0090] In some examples, the I / O module 710 includes a user interface. The user interface can enable a user to interact with the device. In some embodiments, the user interface can include an audio device such as an external speaker system, an external display device such as a display screen, or an input device (e.g., a remote control device that interfaces with the user interface directly or through an I / O controller module). In some cases, the user interface can be a graphical user interface (GUI). In some examples, the communication interface operates at the boundary between the communication entity and the channel and can also record and process communications. A communication interface is provided herein to enable a processing system to be coupled to a transceiver (e.g., a transmitter and / or receiver). In some instances, the transceiver is configured to transmit (or send) and receive signals for the communication device via an antenna. The I / O module 710 is a reference Figure 20 Examples of I / O interfaces are described, or include aspects thereof.

[0091] Examples of memory unit 715 include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of memory unit 715 include solid-state memory and a hard disk drive. In some examples, memory unit 715 is used to store computer-readable, computer-executable software including instructions that, when executed, enable the processor to perform the various functions described herein.

[0092] In some cases, the memory unit 715 includes, among other things, a basic input / output system (BIOS) that controls basic hardware or software operations (such as interaction with peripheral components or devices). In some cases, a memory controller operates the memory unit. For example, the memory controller may include a row decoder, a column decoder, or both. In some cases, the memory cells within the memory unit 715 store information in the form of logical states.

[0093] In one aspect, the memory unit 715 includes the machine learning model 720, the conditional encoder 725, and the image generation model 730. In certain aspects, the machine learning model 720 includes the conditional encoder 725 and the image generation model 730. The memory unit 715 is a reference Figure 20 Examples of memory subsystems are described, or include aspects thereof.

[0094] In some cases, machine learning model 720 is a computing algorithm, model, or system designed to recognize patterns, make predictions, or perform specific tasks (e.g., image processing) without being explicitly programmed. According to certain aspects, machine learning model 720 is implemented as software, firmware, one or more hardware circuits, or a combination thereof stored in memory unit 715 and executable by processor unit 705.

[0095] According to some embodiments of the present disclosure, the machine learning model 720 includes an ANN, which is a hardware or software component comprising a plurality of connected nodes (e.g., artificial neurons) that loosely correspond to neurons in the human brain. Each connection or edge sends a signal from one node to another (like a physical synapse in the brain). When a node receives a signal, the node processes the signal and then sends the processed signal to other connected nodes. In some cases, the signals between nodes include real numbers, and the output of each node is calculated by a function of the sum of its inputs. In some instances, the node can use other mathematical algorithms (e.g., selecting the maximum value from the input as the output) or any other suitable algorithm for activating the node to determine the output. Each node and edge is associated with one or more node weights that determine how to process and send signals.

[0096] During the training process, one or more node weights are adjusted to improve the accuracy of the results (e.g., by minimizing a loss function that corresponds in some way to the difference between the current result and the target result). The weights of the edges increase or decrease the strength of the signal transmitted between the nodes. In some cases, the nodes have a threshold below which no signal is sent at all. In some examples, the nodes are grouped into layers. Different layers perform different transformations on their respective inputs. The initial layer is called the input layer, and the last layer is called the output layer. In some cases, the signal passes through some layers multiple times.

[0097] According to some embodiments, the machine learning model 720 includes a computer-implemented convolutional neural network (CNN). CNN is a class of neural networks commonly used in computer visualization or image classification systems. In some cases, CNN can enable processing of digital images with minimal preprocessing. CNN can be characterized by using convolutional (or cross-correlation) hidden layers. These layers apply a convolution operation to the input before sending the result to the next layer. Each convolution node can process data from a finite field (e.g., a receptive field) of the input. During the forward pass of the CNN, the filters of each layer can be convolved across the input volume, computing the dot product between the filter and the input. During the training process, the filters can be modified so that they activate when they detect specific features in the input.

[0098] In one aspect, the machine learning model 720 includes machine learning parameters. Machine learning parameters, also known as model parameters or weights, are variables that provide the behavior and characteristics of the machine learning model 720. Machine learning parameters can be learned or estimated from training data and used to make predictions or perform tasks based on learned patterns and relationships in the data.

[0099] During the training process, machine learning parameters are adjusted to minimize a loss function or maximize a performance metric. The goal of the training process is to find the optimal values ​​of the parameters that allow the machine learning model 720 to make accurate predictions or perform well for a given task.

[0100] For example, during training, the algorithm adjusts machine learning parameters to minimize the error or loss between the predicted output and the actual target based on optimization techniques such as gradient descent, stochastic gradient descent, or other optimization algorithms. Once the machine learning parameters are learned from the training data, the machine learning parameters are used to make predictions on new, unseen data.

[0101] According to some embodiments, the machine learning model 720 includes a computer-implemented recurrent neural network (RNN). RNN is a type of ANN in which the connections between nodes form a directed graph along an ordered (e.g., time) sequence. This enables RNN to model temporal dynamic behavior, such as predicting what element should appear next in a sequence. Therefore, RNN is suitable for tasks involving ordered sequences such as text recognition (where words are sorted in sentences). In some cases, RNN includes one or more finite impulse recurrent networks (characterized by nodes forming a directed acyclic graph), one or more infinite impulse recurrent networks (characterized by nodes forming a directed cyclic graph), or a combination thereof.

[0102] According to some embodiments, the machine learning model 720 includes a transformer (or transformer model, or transformer network), where the transformer is a neural network model for natural language processing tasks. The transformer network uses an encoder and a decoder to convert a sequence into another sequence. The encoder and decoder include modules that can be stacked on top of each other multiple times. The module includes multi-head attention and feedforward layers. The input and output (target sentence) are first embedded in an n-dimensional space. The position encoding of different words (for example, giving each word / part in the sequence a relative position, because the sequence depends on the order of its elements) is added to the embedding representation of each word (n-dimensional vector). In some examples, the transformer network includes an attention mechanism, where attention looks at the input sequence and decides which other parts of the sequence are important at each step. The attention mechanism involves a query, a keyword, and a value, represented by Q, K, and V, respectively. Q is a matrix containing the query (a vector representation of a word in the sequence), K is the keyword (a vector representation of the word in the sequence), and V is the value, which is also a vector representation of the word in the sequence. For the encoder and decoder, multi-head attention module, V consists of the same word sequence as Q. However, for the attention module that considers encoder and decoder sequences, V is different from the sequence represented by Q. In some cases, the values ​​in V are multiplied with some attention weights a and summed.

[0103] In the field of machine learning, an attention mechanism (e.g., implemented in one or more ANNs) is a method of setting different levels of importance on different elements of the input. Calculating attention can include three basic steps. First, the similarity between the query and the key vector obtained from the input is calculated to generate attention weights. Similarity functions used for this process can include dot products, concatenation, detectors, etc. Next, the attention weights are normalized using a softmax function. Finally, the attention weights are weighted together with the corresponding values. In the context of attention networks, keys and values ​​are vectors or matrices used to represent the input data. The key is used to determine which parts of the input the attention mechanism should focus on, while the value is used to represent the actual data being processed.

[0104] Attention mechanisms are key components in some ANN architectures, particularly those employed in natural language processing (NLP) and sequence-to-sequence tasks, allowing the ANN to focus on different parts of the input sequence when making predictions or generating outputs. Some sequence models (such as RNNs) process the input sequence sequentially, maintaining an internal hidden state that captures information from previous steps. However, in some cases, this sequential processing leads to difficulties in capturing long-term dependencies or focusing on specific parts of the input sequence.

[0105] Attention mechanisms address these difficulties by enabling the ANN to selectively focus on different parts of the input sequence, assigning different levels of importance or attention to each part. Attention mechanisms achieve selective focusing by considering the relevance of each input element relative to the current state of the ANN.

[0106] The term "self-attention" refers to a machine learning model in which representations of the input interact with each other to determine attention weights for the input. Self-attention can be distinguished from other attention models because the attention weights are at least partially determined by the input itself.

[0107] According to some aspects, the machine learning model 720 obtains a structure input indicating a target spatial structure. In some examples, the machine learning model 720 encodes each component in a component set of the structure input to obtain a component structure encoding set, where each component in the component set includes a different representation of the target spatial structure. In some examples, the machine learning model 720 combines the component structure encoding sets to obtain a preliminary structure encoding.

[0108] In some examples, machine learning model 720 provides a structural encoding to a first layer of an image generation model. In some examples, machine learning model 720 obtains structural adhesion parameters, wherein a composite image is generated based on the structural adhesion parameters using the structural encoding. In some examples, machine learning model 720 obtains a textual cue describing an object, wherein a composite image is generated based on the textual cue.

[0109] In some examples, the machine learning model 720 obtains a style hint indicating a style element, wherein a composite image is generated based on the style hint to include the style element. In some examples, the machine learning model 720 obtains a preliminary image. In some examples, the machine learning model 720 generates a structural input based on the preliminary image. The machine learning model 720 is a reference Figures 3 to 5 and Figure 8 Examples of corresponding elements are described or include aspects thereof.

[0110] According to some aspects, the machine learning model 720 receives a structural input indicating a spatial structure. In some examples, the machine learning model 720 receives a set of structural inputs. In some examples, the machine learning model 720 receives structural adhesion parameters, wherein a structural encoding is used to generate a composite image based on the structural adhesion parameters.

[0111] In some examples, the machine learning model 720 obtains textual cues, wherein the composite image is generated based on the textual cues. In some examples, the machine learning model 720 obtains style cues, wherein the composite image is generated based on the style cues.

[0112] According to some aspects, the conditional encoder 725 is implemented as software, firmware, one or more hardware circuits, or a combination thereof that is stored in the memory unit 715 and can be executed by the processor unit 705. According to some aspects, the conditional encoder 725 encodes the structure input to obtain a structure encoding representing the target space structure. In some examples, the conditional encoder 725 encodes the preliminary structure encoding to obtain the structure encoding. In some aspects, each structure encoding in the component structure encoding set is generated by a different structure encoder. In some aspects, each structure encoding in the component structure encoding has a different number of channels.

[0113] In some aspects, the component structure encoding set includes a depth encoding, an edge encoding, and an entity encoding. In some examples, the conditional encoder 725 downsamples the structure encoding to obtain a downsampled structure encoding. In some examples, the conditional encoder 725 provides the downsampled structure encoding to the second layer of the image generation model 730. In some examples, the conditional encoder 725 performs multiple convolutions on the structure encoding.

[0114] According to some aspects, the conditional encoder 725 generates a structured encoding based on the structured input. In some aspects, the conditional encoder 725 includes a set of convolutional layers and a set of activation layers. The conditional encoder 725 is a reference Figure 8 Examples of corresponding elements are described or include aspects thereof.

[0115] According to some aspects, the conditional encoder 725 encodes the structure input to obtain a structure code. In some examples, the conditional encoder 725 encodes each structure input in the structure input set to obtain a structure code set, wherein a composite image is generated based on the structure code set. In some aspects, each structure code in the structure code set is generated by a different structure encoder. In some aspects, each structure code in the structure code set has a different number of channels. In some aspects, the group of structure codes includes depth coding, edge coding, and entity coding. In some examples, the conditional encoder 725 performs multiple convolutions on the structure code. According to some aspects, the conditional encoder 725 includes parameters stored in at least one memory and trained to encode the structure input to obtain the structure code, wherein the structure input indicates a spatial structure. In some aspects, each layer of the conditional encoder 725 includes a convolutional layer or an activation layer.

[0116] In some aspects, the image generation model 730 is implemented as software, firmware, one or more hardware circuits, or a combination thereof, stored in the memory unit 715 and executable by the processor unit 705. In some aspects, the image generation model 730 generates a synthetic image based on structure encoding, wherein the synthetic image depicts an object having a target spatial structure. In some examples, the image generation model 730 obtains a noisy input. In some examples, the image generation model 730 denoises the noisy input based on the structure encoding. In some aspects, the image generation model 730 is trained using a training set of training structure inputs indicating spatial structure and ground truth images including the target spatial structure.

[0117] According to some aspects, the image generation model 730 generates a predicted image based on the structure encoding. According to some aspects, the image generation model 730 generates a synthetic image based on the structure encoding, wherein the synthetic image depicts an object having a target spatial structure. In some aspects, the image generation model 730 includes more parameters than the conditional encoder 725. The image generation model 730 is a reference Figures 9 to 11 and Figure 17 The image generation model 730 is a reference to the corresponding elements of the description or includes aspects thereof. Figure 12 Examples of diffusion models described, or including references Figure 12 Describe various aspects of the diffusion model.

[0118] According to some aspects, image generation model 730 generates a synthetic image based on the structure encoding, wherein the synthetic image depicts an object having spatial structure. In some examples, image generation model 730 provides the structure encoding to a first layer of image generation model 730. In some examples, image generation model 730 downsamples the structure encoding to obtain a downsampled structure encoding. In some examples, image generation model 730 provides the downsampled structure encoding to a second layer of image generation model 730. In some examples, image generation model 730 combines the structure encoding with a noise input to obtain a modified noise input, wherein the synthetic image is generated based on the modified noise input.

[0119] According to some aspects, image generation model 730 includes parameters stored in at least one memory and trained to generate synthetic images based on structure encoding, wherein the synthetic images depict objects having spatial structure. In some aspects, image generation model 730 includes at least ten times more parameters than conditional encoder 725.

[0120] According to some aspects, the training component 735 is implemented as software, firmware, one or more hardware circuits, or a combination thereof stored in the memory unit 715 and executable by the processor unit 705. According to some embodiments, the training component 735 is implemented as software stored in the memory unit and executable by a processor in the processor unit of a separate computing device, firmware in a separate computing device, one or more hardware circuits of a separate computing device, or a combination thereof. In some examples, the training component 735 is part of another device other than the image processing device 700 and communicates with the image processing device 700. In some examples, the training component 735 is part of the image processing device 700.

[0121] According to some aspects, training component 735 obtains a training set comprising a training structure input indicating a spatial structure and a ground truth image comprising the spatial structure. In some examples, training component 735 uses the training set to train image generation model 730 to generate a synthetic image based on the structure input, wherein the synthetic image comprises the spatial structure. In some examples, training component 735 jointly trains conditional encoder 725 with image generation model 730. In some examples, training component 735 calculates a loss function by comparing a predicted image to a ground truth image. In some examples, training component 735 updates parameters of image generation model 730 based on the loss function.

[0122] According to some aspects, the training component 735 obtains a training set including a training structure input and a ground truth image. In some examples, the training component 735 uses the training set to train the machine learning model 720 to generate a synthetic image based on the structure input. In some examples, the training component 735 jointly trains the conditional encoder 725 and the image generation model 730.

[0123] Figure 8 An example of a machine learning model 800 according to aspects of the present disclosure is shown. The example shown includes the machine learning model 800, a structure input 805, a feature encoder 810, a structure encoding 815, a combined structure encoding 820, a conditional encoder 825, a conditional encoding layer 830, a downsampled encoding 835, a noise input 840, an image generation model 842, a convolutional layer 845, downsampled features 850, a combined downsampled features 855, an upsampled layer 860, and output features 865. In one aspect, the machine learning model 800 includes the feature encoder 810, the conditional encoder 825, and the image generation model 842. In one aspect, the conditional encoder 825 includes the conditional encoding layer 830. In one aspect, the image generation model 842 includes one or more convolutional layers 845 and one or more upsampled layers 860.

[0124] Reference Figure 8 , the machine learning model 800 receives an input image and a text prompt to generate a composite image. For example, a feature extractor extracts one or more feature maps from the input image. In some cases, the feature maps are used as structural input 805. In some cases, the feature extractor includes a depth model, an edge model, or an entity segmentation model. In some cases, the structural input 805 includes a depth map, an edge map, and an entity map. For example, in some cases, the dimensions of the depth map can be 1×1024×1024. For example, the dimensions of the edge map can be 2×1024×1024. For example, the dimensions of the entity map can be 3×1024×1024.

[0125] Each of the structure inputs is provided to a feature encoder 810, where the feature encoder 810 encodes the structure input 805 to obtain a structure encoding 815. In some cases, for example, the feature encoder 810 includes a depth encoder, an edge encoder, or an entity encoder. In one aspect, the feature encoder 810 includes one or more convolutional blocks and / or one or more downsampling layers. In some cases, the structure encoding 815 has a smaller dimension than the structure input 805. For example, the structure encoding 815 can have a dimension of 3×1024×1024. In some cases, the structure encoding 815 for each structure input of different structure inputs has the same number of channels (e.g., three channels).

[0126] In some embodiments, each structure code in the structure code is combined to form a combined structure code 820. In one embodiment, the noise input 840 is combined with the structure code to generate the combined structure code 820. The combined structure code 820 is then input to the conditional encoder 825. For example, the combined structure code 820 passes through the conditional coding layer 830 of the conditional encoder 825. In some cases, the conditional coding layer 830 outputs a downsampled code 835 (or a first downsampled code). In one aspect, the conditional coding layer 830 includes one or more convolutional layers or one or more activation layers. For example, the activation layer is an S-shaped activation layer that uses a sigmoid function to map the input to a range between 0 and 1, where the output can be interpreted as a probability.

[0127] In some embodiments, the noise input 840 is input to a U-Net of an image generation model 842 (e.g., Figure 13 The convolutional layer 845 of the U-Net described in the embodiment of the present invention is used to generate a downsampled feature 850. In response, the convolutional layer 845 generates a downsampled feature 850. For example, the downsampled feature 850 has the same dimensions as the noise input 840 (e.g., 128×128). In some cases, the number of channels is expanded in the downsampled feature 850 (e.g., from 12 to 320). The downsampled feature 850 is then combined with the downsampled code 835 from the conditional encoder 825 to generate a combined downsampled feature 855 (e.g., a first combined downsampled feature).

[0128] In some embodiments, the downsampling process is repeated multiple times (e.g., three times in total). For example, the first downsampling code is input to the conditional coding layer 830 (e.g., the second conditional coding layer) to generate a second downsampling code, wherein the dimension of the second downsampling code is reduced by half (e.g., from 128×128 to 64×64). Similarly, the first combined downsampling feature is input to the second convolutional layer to generate the second downsampling feature. The dimension of the second downsampling feature is reduced by half (e.g., from 128×128 to 64×64). Then, the second downsampling code generated by the conditional encoder 825 and the second downsampling feature generated by the image generation model 842 are combined to form a second combined downsampling feature.

[0129] In some embodiments, after the downsampling process, the combined downsampled features 855 are fed to the upsampling layer 860 of the U-Net of the image generation model 842. In some cases, the upsampling layer 860 includes the same number of upsampling layers as the number of downsampling layers. For example, the U-Net architecture includes two downsampling layers and two upsampling layers. In some cases, the U-Net architecture includes an intermediate layer between the downsampling layers and the upsampling layers. In the final upsampling step, the upsampling layer 860 generates output features 865 based on the combined downsampled features 855. In some embodiments, the output features 865 are input to the image decoder (e.g., FIG. Figure 12 ) to generate a composite image.

[0130] According to some embodiments, the textual prompt is used as guidance for the image generation model 842. For example, the textual prompt can be encoded using a text encoder to obtain guidance features in a guidance space. The guidance features can be combined with noise features (e.g., noise input 840) at one or more layers of the back-diffusion process so that the synthesized image includes the content described by the textual prompt. For example, the guidance features from the textual prompt are combined with the noise input 840 using a cross-attention block during the back-diffusion process of the image generation model 842.

[0131] In some embodiments, the image generation model 842 receives a style hint describing a style to generate a composite image including elements corresponding to the style described by the style hint. In some embodiments, the image generation model 842 includes multiple levels of structural adhesion.

[0132] According to some aspects, the image generation model 842 of the present disclosure includes a U-Net architecture with residual connections. For example, the U-Net architecture can be divided into three blocks: an input block, an intermediate block, and an output block. In some cases, the input block includes one or more convolutional layers 845. In some cases, the intermediate block includes an intermediate layer. In some cases, the output block includes one or more upsampling layers 860. For example, the input block includes multiple blocks and a downsampling layer after every "n" number of blocks.

[0133] In some cases, the machine learning model 800 receives one or more of three input conditions (e.g., structural input 805): depth, edge, or entity. The input conditions are spatial conditions with the shape of the original image and with 1, 2, and 3 channels, respectively. For each input condition, the feature encoder 810 includes a lightweight network consisting of ResBlocks and downsampling layers, where the output of the structural encoder has the same shape as the input potential of the U-Net (e.g., noise input 840).

[0134] In some embodiments, the condition is concatenated channel by channel to the noise input (e.g., noise input 840) to obtain a combined code (e.g., combined structural code 820). The combined code is passed through a set of convolutional blocks and a downsampling layer. In one embodiment, the combined code is downsampled three times to obtain intermediate outputs, which are stored. The intermediate outputs are then added after the three downsampling layers in the U-Net. Thus, the U-Net learns to obey the input condition.

[0135] Machine Learning Model 800 is a reference Figures 3 to 5 and Figure 7 The conditional encoder 825 is a reference to the corresponding element of the description or includes aspects thereof. Figure 7 Examples of corresponding elements are described or include aspects thereof.

[0136] Figure 9 An example of image generation based on a reference image according to aspects of the present disclosure is shown. The example shown includes an input image 900, a depth model 905, a depth map 910, an edge model 915, an edge map 920, a text hint 925, an image generation model 930, and a synthesized image 935.

[0137] Reference Figure 9 , the image generation model 930 receives the input image 900 and generates a composite image 935. For example, the input image 900 is provided to the depth model 905 to generate a depth map 910. Alternatively, the input image 900 is provided to the edge model 915 to generate an edge map 920. In one embodiment, the image generation model 930 receives the depth map 910 and the text hint 925 to generate the composite image 935. In one embodiment, the image generation model 930 receives the edge map 920 and the text hint 925 to generate the composite image 935. In one embodiment, the image generation model 930 receives the depth map 910, the edge map 920, and the text hint 925 to generate the composite image 935.

[0138] The input image 900 is a reference Figures 3 to 5 and Figure 10 Examples of corresponding elements described herein, or including aspects thereof. Depth model 905 is a reference Figure 17 Depth map 910 is a reference to the corresponding element of the example or including aspects thereof. Figure 3 、 Figure 5 and Figure 17 Examples of corresponding elements are described or include aspects thereof.

[0139] Edge map 920 is a reference Figure 4 、 Figure 5 and Figure 17 Examples of corresponding elements described herein, or including aspects thereof. Text hint 925 is a reference to Figures 3 to 5 、 Figure 10 、 Figure 11 and Figure 17 The image generation model 930 is a reference to the corresponding elements of the description or includes aspects thereof. Figure 7 、 Figure 10 、 Figure 11 and Figure 17 Examples of corresponding elements described herein or including aspects thereof. Composite image 935 is a reference Figure 10 and Figure 11 Examples of corresponding elements are described or include aspects thereof.

[0140] Figure 10 An example of entity-controlled image generation according to aspects of the present disclosure is shown. The example shown includes an input image 1000 , an entity segmentation model 1005 , an entity map 1010 , textual prompts 1015 , an image generation model 1020 , and a composite image 1025 .

[0141] Reference Figure 10 , the image generation model 1020 receives the input image 1000 and generates a composite image 1025 based on the input image 1000. For example, the input image 1000 is provided to the entity segmentation model 1005 to generate an entity map 1010. In some cases, the entity map 1010 includes one or more pixels representing objects within the scene of the input image 1000. The image generation model 1020 receives the entity map 1010 and the text prompt 1015 to generate the composite image 1025.

[0142] The input image 1000 is the reference Figures 3 to 5 and Figure 9 The entity segmentation model 1005 is a reference to the corresponding elements of the description or includes aspects thereof. Figure 17 Examples of corresponding elements described herein or including aspects thereof. Entity diagram 1010 is a reference to Figure 17 Examples of corresponding elements are described or include aspects thereof.

[0143] Text prompt 1015 is a reference Figures 3 to 5 、 Figure 9 、 Figure 11 and Figure 17 The image generation model 1020 is a reference to the corresponding elements of the description or includes aspects thereof. Figure 7 、 Figure 9 、 Figure 11 and Figure 17 Examples of corresponding elements described herein or including aspects thereof. Composite image 1025 is a reference Figure 9 and Figure 11 Examples of corresponding elements are described or include aspects thereof.

[0144] Figure 11 An example of user sketch-based image generation according to aspects of the present disclosure is shown. The example shown includes a sketch input 1100, a text prompt 1105, an image generation model 1110, and a composite image 1115.

[0145] Reference Figure 11 , the image generation model 1110 receives the sketch input 1100 and the text prompt 1105 to generate a composite image 1115. In some cases, the system can generate one or more of the structural inputs based on the sketch input 1100. For example, the structural input includes one or more of a combination of a depth map, an edge map, and a solid map. The image generation model 1110 uses a reference image to generate a composite image 1115 based on the structural input and the text prompt 1105. Figure 8 The image generation process described is used to generate the composite image 1115.

[0146] Text prompt 1105 is a reference Figures 3 to 5 、 Figure 9 、 Figure 10 and Figure 17 The image generation model 1110 is a reference to the corresponding elements of the description or includes aspects thereof. Figure 7 、 Figure 9 、 Figure 10 and Figure 17 Examples of corresponding elements described herein or including aspects thereof. Composite image 1115 is a reference Figure 9 and Figure 10 Examples of corresponding elements are described or include aspects thereof.

[0147] Figure 12 An example of an image generation model according to aspects of the present disclosure is shown. The example shown includes a diffusion model 1200, an original image 1205, a pixel space 1210, an image encoder 1215, original image features 1220, a latent space 1225, a forward diffusion process 1230, noise features 1235, a backward diffusion process 1240, denoised image features 1245, an image decoder 1250, an output image 1255, a textual prompt 1260, a textual encoder 1265, guidance features 1270, and a guidance space 1275.

[0148] Diffusion models are a class of generative neural networks that can be trained to generate new data with features similar to those found in the training data. In particular, diffusion models can be used to generate new images. Diffusion models can be used for a variety of image generation tasks, including image super-resolution, generation of images with perceptual metrics, conditional generation (e.g., generation based on textual guidance, color guidance, style guidance, and image guidance), image inpainting, and image manipulation.

[0149] Types of diffusion models include denoising diffusion probabilistic models (DDPMs) and denoising diffusion implicit models (DDIMs). In DDPMs, the generation process involves inverting a random Markov diffusion process. DDIMs, on the other hand, use a deterministic process so that the same input generates the same output. Diffusion models can also be characterized by whether the noise is added to the image or to image features generated by the encoder (e.g., latent diffusion).

[0150] The diffusion model works by iteratively adding noise to the data during the forward process and then learning to recover the data by denoising the data during the backward process. For example, during training, the diffusion model 1200 may take as input an original image 1205 in pixel space 1210 and apply an image encoder 1215 to convert the original image 1205 into original image features 1220 in a latent space 1225. Then, a forward diffusion process 1230 gradually adds noise to the original image features 1220 to obtain noise features 1235 (also in the latent space 1225) at various noise levels.

[0151] Next, a back diffusion process 1240 (e.g., a U-Net ANN) gradually removes noise from the noise features 1235 at various noise levels to obtain denoised image features 1245 in the latent space 1225. In some examples, the denoised image features 1245 are compared with the original image features 1220 at each of the various noise levels, and the parameters of the back diffusion process 1240 of the diffusion model are updated based on the comparison. Finally, the image decoder 1250 decodes the denoised image features 1245 to obtain an output image 1255 in the pixel space 1210. In some cases, the output image 1255 is generated at each of the various noise levels. The output image 1255 can be compared with the original image 1205 to train the back diffusion process 1240. In some cases, the output image 1255 refers to a synthesized image (e.g., a reference image). Figures 3 to 5 described).

[0152] In some cases, the image encoder 1215 and the image decoder 1250 are pre-trained before training the backdiffusion process 1240. In some examples, the image encoder 1215 and the image decoder 1250 are jointly trained, or the image encoder 1215 and the image decoder 1250 are jointly fine-tuned with the backdiffusion process 1240.

[0153] The back diffusion process 1240 can also be guided based on the textual cue 1260 or another guiding cue such as an image, layout, style, color, segmentation map, etc. The textual cue 1260 can be encoded using a text encoder 1265 (e.g., a multimodal encoder) to obtain guidance features 1270 in a guidance space 1275. The guidance features 1270 can be combined with the noise features 1235 at one or more layers of the back diffusion process 1240 to ensure that the output image 1255 includes the content described by the textual cue 1260. For example, the guidance features 1270 can be combined with the noise features 1235 using a cross-attention block within the back diffusion process 1240.

[0154] Cross-attention, also known as multi-head attention, is an extension of the attention mechanism used in some ANNs, for example, for NLP tasks. In some cases, cross-attention attends to multiple parts of the input sequence simultaneously, capturing interactions and dependencies between different elements. In cross-attention, there are two input sequences: a query sequence and a key-value sequence. The query sequence represents the elements to be attended to, while the key-value sequence includes the elements to be focused on. In some cases, to compute cross-attention, the cross-attention block transforms (e.g., using linear projection) each element in the query sequence into a "query" representation, and transforms the elements in the key-value sequence into "key" and "value" representations.

[0155] The cross-attention block calculates an attention score by measuring the similarity between each query representation and the key representation, where a higher similarity indicates that more attention is paid to the key element. The attention score indicates the importance or relevance of each key element to the corresponding query element.

[0156] The cross-attention block then normalizes the attention scores to obtain attention weights (e.g., using a softmax function), where the attention weights determine how much information from each value element is incorporated into the final attention representation. By simultaneously focusing on different parts of the key-value sequence, the cross-attention block captures the relationships and dependencies between the input sequences, enabling the machine learning model to understand the context and generate more accurate and context-relevant outputs.

[0157] In some examples, the diffusion model is based on a neural network architecture called U-Net. U-Net takes input features with an initial resolution and an initial number of channels and processes the input features using an initial neural network layer (e.g., a convolutional network layer) to generate intermediate features. The intermediate features are then downsampled using a downsampling layer so that the downsampled features have a resolution less than the initial resolution and a number of channels greater than the initial number of channels.

[0158] This process is repeated multiple times, and then reversed. For example, the downsampled features are upsampled using the upsampling process to obtain upsampled features. The upsampled features can be combined with intermediate features of the same resolution and number of channels via skip connections. These inputs are processed using the final neural network layer to generate output features. In some cases, the output features have the same resolution as the initial resolution and the same number of channels as the initial number of channels.

[0159] In some cases, U-Net takes additional input features to generate conditionally generated outputs. For example, the additional input features can include vector representations of input prompts. The additional input features can be combined with intermediate features within the neural network at one or more layers. For example, a cross-attention module can be used to combine additional input features with intermediate features. Figure 13 Describe U-Net in more details.

[0160] The diffusion process can also be modified based on conditional guidance. In some cases, for example, the user provides a text prompt (e.g., text prompt 1260) that describes the content included in the generated image. In some examples, the guidance can be provided in a form other than text, such as via an image, sketch, color, style, or layout. The system converts the text prompt 1260 (or other guidance) into a conditional guidance vector or other multidimensional representation. For example, a transformer model or a multimodal encoder can be used to convert the text into a vector or a series of vectors. In some cases, the encoder for conditional guidance is trained independently of the diffusion model.

[0161] Initialize a noise map comprising random noise. The noise map can be in pixel space or latent space. By initializing the image with random noise, different variations of the image can be generated that include the content described by the conditional guidance. Then, the diffusion model 1200 generates an image based on the noise map and the conditional guidance vector.

[0162] The diffusion process may include both a forward diffusion process 1230 for adding noise to an image (e.g., original image 1205) or a feature (e.g., initial image feature 1220) in the latent space 1225, and a backward diffusion process 1240 for denoising the image (or feature) to obtain a denoised image (e.g., output image 1255). The forward diffusion process 1230 may be represented as q(x t |x t-1 ), the reverse diffusion process 1240 can be represented as p θ (x t-1 |x t ). Reference Figure 14 Describe the diffusion process in more detail.

[0163] Diffusion model 1200 can be trained using both forward diffusion process 1230 and backward diffusion process 1240. In one example, an untrained model is initialized. Initialization can include defining the architecture of the model and establishing initial values ​​for model parameters. In some cases, initialization can include defining hyperparameters such as the number of layers, the resolution and channels of each layer block, the location of skip connections, etc.

[0164] The system then adds noise to the training image using N stages of forward diffusion 1230. In some cases, the forward diffusion 1230 is a fixed process in which Gaussian noise is sequentially added to the image. In a latent diffusion model, Gaussian noise can be sequentially added to features in the latent space 1225 (e.g., original image features 1220).

[0165] At each stage n, starting from stage N, a backward diffusion process 1240 is used to predict the image or image features of stage n-1. For example, the backward diffusion process 1240 can predict the noise added by the forward diffusion process 1230 and the predicted noise can be removed from the image to obtain a predicted image. In some cases, the original image 1205 is predicted at each stage of the training process.

[0166] Training components (e.g., reference Figure 7 The training component described herein compares the predicted image (or image features) at stage n-1 with the actual image (or image features), such as the image at stage n-1 or the original input image. For example, given observation data x, the diffusion model 1200 can be trained to minimize the negative log-likelihood of the training data - log p θ The training component then updates the parameters of the diffusion model 1200 based on the comparison. For example, gradient descent can be used to update the parameters of the U-Net. The time-dependent parameters of the Gaussian transformation can also be learned. Figure 19 Describes more details about training diffusion models.

[0167] Figure 13 An example of a U-Net 1300 according to aspects of the present disclosure is shown. The example shown includes the U-Net 1300, input features 1305, an initial neural network layer 1310, intermediate features 1315, a downsampling layer 1320, downsampled features 1325, an upsampling process 1330, upsampled features 1335, skip connections 1340, a final neural network layer 1345, and output features 1350.

[0168] In some examples, U-Net 1300 is the execution reference Figure 12 Examples of components of the reverse diffusion process 1240 of the diffusion model 1200 are described and include reference to Figure 7 Architectural elements of the image generation model 730 are described. Figure 13 The U-Net 1300 depicted in the reference Figure 12 Examples of architectures used in the backdiffusion process are described, or aspects thereof are included.

[0169] In some examples, the diffusion model is based on a neural network architecture called U-Net. U-Net 1300 takes input features 1305 having an initial resolution and an initial number of channels and processes the input features 1305 using an initial neural network layer 1310 (e.g., a convolutional network layer) to generate intermediate features 1315. The intermediate features 1315 are then downsampled using a downsampling layer 1320 so that the downsampled features 1325 have a resolution less than the initial resolution and a number of channels greater than the initial number of channels.

[0170] This process is repeated multiple times, and then the process is reversed. For example, the downsampled features 1325 are upsampled using the upsampling process 1330 to obtain upsampled features 1335. The upsampled features 1335 can be combined with the intermediate features 1315 having the same resolution and number of channels via skip connections 1340. These inputs are processed using the final neural network layer 1345 to generate output features 1350. In some cases, the output features 1350 have the same resolution as the initial resolution and the same number of channels as the initial number of channels.

[0171] In some cases, U-Net 1300 employs additional input features to generate conditionally generated outputs. For example, the additional input features may include vector representations of input prompts. The additional input features may be combined with intermediate features 1315 within the neural network at one or more layers. For example, a cross-attention module may be used to combine the additional input features with intermediate features 1315.

[0172] Diffusion process

[0173] Figure 14 An example of a diffusion process 1400 according to aspects of the present disclosure is shown. The example shown includes the diffusion process 1400, a forward diffusion process 1405, a backward diffusion process 1410, a noise image 1415, a first intermediate image 1420, a second intermediate image 1425, and an original image 1430.

[0174] The diffusion process 1400 may include a forward diffusion process 1405 for diffusing the original image 1430 (e.g., reference image 1430) into the latent space. Figure 12 The original image 1205 described) or features (e.g., reference Figure 12In some aspects, the diffusion process 1400 includes a forward diffusion process 1410 for denoising the noisy image 1415 (or image features) to obtain a denoised image (or original image 1430). The forward diffusion process 1405 can be represented as q(x t |x t-1 ), the reverse diffusion process 1410 can be expressed as p θ (x t-1 |x t In some cases, a forward diffusion process 1405 is used during training to generate images with successively greater noise, and the neural network is trained to perform a backward diffusion process 1410 (e.g., to successively remove noise).

[0175] In potential diffusion models (e.g., ref. Figure 12 In the example forward diffusion process 1405 of the diffusion model 1200 described above, the diffusion model uses a Markov chain to map the observed variable x0 (in pixel space or latent space) to obtain intermediate variables x1, ..., x T When the latent variables pass through a neural network such as U-Net, the Markov chain gradually adds Gaussian noise to the data to obtain an approximate posterior q(x 1：T |x0), where x1,…,x T Has the same dimensions as x0.

[0176] The neural network can be trained to perform the back diffusion process 1410. In the back diffusion process 1410, the diffusion model is trained from the noise data x T Starting with a noisy image 1415, and denoising the data to obtain p θ (x t-1 |x t ). At each step t-1, the back diffusion process 1410 converts x t , such as the first intermediate image 1420 and t as input. Here, t represents a step in the transformation sequence associated with different noise levels. The back diffusion process 1410 iteratively outputs x t-1 , such as the second intermediate image 1425, until x t Restore to x0, that is, the original image 1430. The back diffusion process 1410 can be expressed as:

[0177] p θ (x t-1 |x t ):=N(x t-1 ;μ θ (x t , t), ∑ θ (x t,t)) (#)

[0178] The joint probability of a sample sequence in a Markov chain can be written as the product of the conditional probability and the marginal probability:

[0179]

[0180] where p(x T )=N(x T 0, I) is a pure noise distribution because the backward diffusion process 1410 takes the result of the forward diffusion process 1405 and a sample of pure noise as input, and represents the sequence of Gaussian transformations corresponding to the sequence with Gaussian noise added to the samples.

[0181] At the interference time, the observed data x0 in the pixel space can be mapped to the latent space as input, and the generated data x is mapped back from the latent space to the pixel space as output. In some examples, x0 represents the original input image with low image quality, and the latent variables x1, ..., x T represents a noisy image, and x represents a generated image with high image quality.

[0182] Image processing

[0183] Figure 15 An example of a method 1500 for generating a composite image using a structured encoder according to aspects of the present disclosure is shown. In some examples, these operations are performed by a system including a processor that executes a set of codes to control functional elements of the device. Additionally or alternatively, certain processes are performed using dedicated hardware. Generally, these operations are performed according to the methods and processes described according to aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in conjunction with other operations.

[0184] At operation 1505, the system encodes each component in the component set of the structure input to obtain a component set structure encoding, wherein each component in the component set includes a different representation of the target space structure. In some cases, the operation of this step refers to or can be referred to by Figure 8 In some cases, the components of the structural input can be referred to as references. Figure 8 The feature maps described herein include, for example, depth maps, edge maps, and / or entity maps. In some cases, each feature map of the feature maps represents a different target spatial structure of the input image.

[0185] At operation 1510, the system combines the component structure code sets to obtain preliminary structure codes. In some cases, the operation of this step involves or can be performed by reference Figure 7 and Figure 8In some cases, for example, preliminary structural encoding is performed in conjunction with Figure 8 In some cases, the preliminary structural encoding comprises one or more structural codes and a noise input.

[0186] At operation 1515, the system encodes the preliminary structure code using the conditional encoder to obtain the structure code. In some cases, the operation of this step involves or can be performed by reference Figure 7 and Figure 8 In some cases, the structural encoding can be referred to as Figure 8 For example, the downsampled code (or structured code) is combined with the downsampled features generated by the image generation model to obtain a combined downsampled code.

[0187] Training and evaluation

[0188] exist Figures 16 to 19 In the present invention, a method, apparatus, non-transitory computer-readable medium, and system for training a machine learning model include: obtaining a training set including a training structure input indicating a spatial structure and a reference truth image including the spatial structure; and using the training set to train an image generation model to generate a synthetic image based on the structure input, wherein the synthetic image includes the spatial structure.

[0189] Some examples of the methods, apparatus, non-transitory computer-readable media, and systems also include jointly training the conditional encoder with the image generation model. In some aspects, the image generation model is trained using a training set comprising a training structure input indicating spatial structure and a ground truth image comprising the spatial structure.

[0190] Some examples of methods, apparatuses, non-transitory computer-readable media, and systems further include generating a structured encoding based on the structured input. Some examples further include generating a predicted image based on the structured encoding. Some examples further include calculating a loss function by comparing the predicted image to a ground truth image. Some examples further include updating parameters of an image generation model based on the loss function.

[0191] According to certain aspects, methods, apparatuses, non-transitory computer-readable media, and systems for training a machine learning model are described. One or more aspects of the methods, apparatuses, non-transitory computer-readable media, and systems include obtaining a training set comprising a training structure input and a ground truth image, and training, using the training set, to generate a synthetic image based on the structure input.

[0192] Figure 16An example of a method 1600 for training a machine learning model according to various aspects of the present disclosure is shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of the device. Additionally or alternatively, dedicated hardware is used to perform certain processes. Generally, these operations are performed according to the methods and processes described according to various aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed together with other operations.

[0193] At operation 1605, the system obtains a training set including a training structure input indicating a spatial structure and a ground truth image including the spatial structure. In some cases, the operation of this step involves or can be performed by reference Figure 7 The training components described above are used to perform the training. In some cases, the training set includes text-image pairs. For example, the training images are fed into a feature extractor to obtain training structure inputs. In some cases, the training images (or ground truth images) include elements of spatial structure.

[0194] At operation 1610, the system trains an image generation model using a training set to generate a synthetic image based on a structure input, wherein the synthetic image includes a spatial structure. In some cases, the operation of this step involves or can be performed by reference Figure 7 In some cases, the training component computes a diffusion loss based on the synthesized image and the ground truth image, wherein the image generation model is updated based on the diffusion loss.

[0195] Figure 17 An example of a graphical depiction according to aspects of the present disclosure is shown. The illustrated example includes a training system 1700, a training image 1705, a feature extractor 1710, a feature map 1735, a training structure input 1760, a text prompt 1765, and an image generation model 1770. In one aspect, the feature extractor 1710 includes a depth model 1715, an edge detection model 1720, a scribble model 1725, and an entity segmentation model 1730. In one aspect, the feature map 1735 includes a depth map 1740, an edge map 1745, a scribble map 1750, and an entity map 1755.

[0196] Reference Figure 17, training image 1705 is provided to feature extractor 1710 to generate feature map 1735. In some cases, for example, feature extractor 1710 includes depth model 1715, edge detection model 1720, scribble model 1725, and entity segmentation model 1730. In some cases, feature map 1735 includes depth map 1740, edge map 1745, scribble map 1750, and entity map 1755. In some cases, each feature map in feature map 1735 is generated based on training image 1705 using a corresponding feature extractor 1710. For example, depth model 1715 generates depth map 1740 based on training image 1705. For example, edge detection model 1720 generates edge map 1745 based on training image 1705. For example, scribble model 1725 generates scribble map 1750 based on training image 1705. For example, entity segmentation model 1730 generates entity map 1755 based on training image 1705.

[0197] According to some embodiments, image generation model 1770 is trained based on feature maps 1735, training structure input 1760, and textual prompts 1765. For example, image generation model 1770 is trained using one or more feature maps 1735 (e.g., depth map 1740, edge map 1745, scribble map 1750, and entity map 1755) and textual prompts 1765 to generate a synthetic image. In some cases, a loss can be calculated based on each feature map 1735 and training structure input 1760. In some cases, a diffusion loss is calculated based on the synthetic image and training image 1705. In some cases, parameters of image generation model 1770 are updated based on the loss, the diffusion loss, or a combination thereof.

[0198] Depth model 1715 is a reference Figure 9 The entity segmentation model 1730 is a reference to the corresponding elements of the description or includes aspects thereof. Figure 10 Depth map 1740 is a reference to the corresponding elements of the description or includes aspects thereof. Figure 3 、 Figure 5 and Figure 9 Examples of corresponding elements are described or include aspects thereof.

[0199] Edge map 1745 is a reference Figure 4 、 Figure 5 and Figure 9 Examples of the corresponding elements described or including aspects thereof. Entity map 1755 is a reference Figure 10 Examples of corresponding elements described herein or including aspects thereof. Text hint 1765 is a reference to Figures 3 to 5 and Figures 9 to 11 Examples of corresponding elements described herein or including aspects thereof. Image generation model 1770 is a reference Figure 7 and Figures 9 to 11 Examples of corresponding elements are described or include aspects thereof.

[0200] Figure 18 An example of a flowchart depicting an algorithm 1800 as a step-by-step process in an example implementation of operations for training a machine learning model according to aspects of the present disclosure is shown. In some embodiments, the algorithm 1800 describes a process as described with reference to FIG. Figure 7 The operations described for configuring the training component 735 of the image generation model 730 and / or the conditional encoder 725. Algorithm 1800 provides one or more examples of generating training data, using the training data to train a machine learning model, and using the trained machine learning model to perform a task.

[0201] In this example, the machine learning system begins by collecting training data (block 1802) to serve as the basis for training a machine learning model that defines the content to be modeled. Training data can be collected by the machine learning system from a variety of sources. Examples of training data sources include public datasets, service provider system platforms that expose application programming interfaces (e.g., social media platforms), user data collection systems (e.g., digital surveys and online crowdsourcing systems), etc. Training data collection can also include data augmentation and synthetic data generation techniques to expand and diversify the available training data, balancing techniques to balance multiple positive and negative examples, and so on.

[0202] The machine learning system can also be configured to identify features relevant to the type of task for which the machine learning model is to be trained (block 1804). Examples of tasks include classification, natural language processing, generative artificial intelligence, recommendation engines, reinforcement learning, clustering, and the like. To this end, the machine learning system collects training data based on the identified features and / or filters the training data based on the identified features after collection. The training data is then used to train the machine learning model.

[0203] To train the machine learning model in the example shown, the machine learning model is first initialized (block 1806). Initialization of the machine learning model includes selecting a model architecture to train (block 1808). Examples of model architectures include neural networks, convolutional neural networks (CNNs), long short-term memory (LSTM) neural networks, generative adversarial networks (GANs), decision trees, support vector machines, linear regression, logistic regression, Bayesian networks, random forest learning, dimensionality reduction algorithms, boosting algorithms, deep learning neural networks, U-Net architectures, and the like.

[0204] A loss function is also selected (block 1810). The loss function is used to measure the difference between the output of the machine learning model (e.g., the model prediction) and the target value used to train the machine learning model (e.g., as expressed by the training data). In addition, an optimization algorithm is selected (block 1812) to be used with the loss function to optimize the parameters of the machine learning model during training, examples of which include gradient descent, stochastic gradient descent (SGD), etc.

[0205] Initialization of the machine learning model also includes setting initial values ​​of the machine learning model (block 1816), examples of which include initializing the weights and biases of the nodes to improve the efficiency of training and the consumption of computing resources as part of the training. Hyperparameters for controlling the training of the machine learning model are also set (block 1814), examples of which include regularization parameters, model parameters (e.g., the number of layers in the neural network), learning rates, batch sizes selected from the training data, etc. Various techniques are used to set hyperparameters, including randomization techniques, heuristics learned from other training scenarios, etc.

[0206] The machine learning system then uses the training data to train a machine learning model (block 1818). A machine learning model refers to a computer representation that can be adjusted (e.g., trained and retrained) based on the input of the training data to approximate an unknown function. In particular, the term machine learning model can include a model that learns and relearns by analyzing the training data using an algorithm (e.g., using the model architecture described above) to generate outputs that reflect the patterns and properties expressed in the training data, thereby learning from known data and making predictions thereon.

[0207] Examples of training types include supervised learning using labeled data, unsupervised learning that involves finding underlying structure or patterns in training data, reinforcement learning based on optimization functions (e.g., rewards and / or penalties), using nodes as part of "deep learning," and the like. For example, a machine learning model can be configured to include multiple nodes that together form multiple layers. For example, these layers can be configured to include an input layer, an output layer, and one or more hidden layers. Computations are performed by the nodes within the layers through hidden states, through a system of weighted connections that are "learned" during training, such as by using a selected loss function and backpropagation to optimize the performance of the machine learning model to perform the associated task.

[0208] As part of training the machine learning model, a determination is made as to whether stopping criteria for validating the machine learning model are met (decision block 1820). The stopping criteria can be used to reduce overfitting of the machine learning model, reduce consumption of computational resources, and improve the ability of the machine learning model to handle unseen data that is not included as examples in the training data. Examples of stopping criteria include, but are not limited to, a predetermined number of time periods, verifying that loss has stabilized, achieving a performance improvement threshold, whether a threshold level of accuracy has been met, or based on performance metrics such as precision and recall. If the stopping criteria have not been met ("No" from decision block 1820), then in this example, the algorithm 1800 continues training the machine learning model using the training data (block 1818).

[0209] If the stopping criteria are met ("yes" from decision block 1820), then the trained machine learning model is used to generate output based on subsequent data (block 1822). For example, the trained machine learning model is trained to perform the task as described above, and thus, once trained, is configured to perform the task based on subsequent data received as input and processed by the machine learning model.

[0210] Figure 19 An example of a method 1900 for training a diffusion model according to aspects of the present disclosure is shown. In some examples, the operations are performed by a system including a processor that executes a set of code to control functional elements of the device. Additionally or alternatively, dedicated hardware is used to perform certain processes. Generally, the operations are performed according to the methods and processes described according to aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in conjunction with other operations.

[0211] In some embodiments, method 1900 describes the Figure 7 The operation of the training component 735 for training the image generation model 730 is described. Method 1900 represents the operation for training the image generation model 730 as described above. Figure 14 In some examples, these operations are performed by a system including a processor that executes a set of codes to control functional elements of an apparatus, such as Figure 7 The image generation model described in .

[0212] At operation 1905, the system initializes the untrained model. In some cases, the operation of this step involves or can be performed by reference Figure 7 Initialization can be performed using the training components described in

[15] . Initialization can include defining the model architecture and establishing initial values ​​for model parameters. In some cases, initialization can include defining hyperparameters such as the number of layers, the resolution and channels of each layer block, the location of skip connections, etc.

[0213] At operation 1910, the system adds noise to the media item using a forward diffusion process in N stages. In some cases, the operation of this step involves or can be performed by reference Figure 7 The training component described above is performed. In some cases, for example, the media items are training images. In some cases, the forward diffusion process is a fixed process in which Gaussian noise is sequentially added to the media items (such as the original images). In a latent diffusion model, Gaussian noise can be sequentially added to features in the latent space.

[0214] At operation 1915, the system predicts the media items for stage n-1 at each stage n starting from stage N. In some cases, the operation of this step involves or can be performed by reference Figure 7 The training components described herein are performed. In some cases, the media item is a synthetic image generated using an image generation model. For example, the back-diffusion process can predict the noise added by the forward diffusion process, and the predicted noise can be removed from the noisy input to obtain a predicted output. In some cases, the original media item is predicted at each training stage of the training process.

[0215] At operation 1920, the system compares the predicted media item (or feature) at stage n-1 with the media at stage n-1. For example, in some cases, the system compares the synthesized image (or predicted image feature) at state n-1 with the ground truth image (or ground truth feature) at state n-1. In some cases, the operation of this step involves or can be performed by reference Figure 7 The training component described performs the following. For example, given observation data x, a diffusion model can be trained to minimize the negative log-likelihood of the training data - log p θ The variational upper bound of (x).

[0216] At operation 1925, the system updates the parameters of the model based on the comparison. In some cases, the operation of this step involves or can be performed by reference Figure 7 The training components described in

[15] can be used to perform this. For example, gradient descent can be used to update the parameters of the U-Net. Time-dependent parameters of the Gaussian transformation can also be learned.

[0217] computing devices

[0218] Figure 20 An example of a computing device 2000 according to aspects of the present disclosure is shown. The example shown includes computing device 2000, processor 2005, memory subsystem 2010, communication interface 2015, I / O interface 2020, user interface component 2025, and channel 2030.

[0219] In some embodiments, the computing device 2000 is a reference Figure 1 and Figure 7 Examples of image processing apparatus described herein or including aspects thereof. In some embodiments, computing device 2000 includes a processor 2005 that can execute instructions stored in memory subsystem 2010 to obtain a structure input indicating a target spatial structure, encode the structure input to obtain a structure code representing the target spatial structure, and generate a composite image based on the structure code.

[0220] According to some embodiments, the processor 2005 includes one or more processors. In some cases, the processor 2005 is an intelligent hardware device (e.g., a general processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or a combination thereof). In some cases, the processor 2005 is configured to operate a memory array using a memory controller. In other cases, the memory controller is integrated into the processor 2005. In some cases, the processor 2005 is configured to execute computer-readable instructions stored in the memory to perform various functions. In some embodiments, the processor 2005 includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing. The processor 2005 is a reference Figure 7 Examples of or including aspects of a processor unit are described.

[0221] According to some embodiments, the memory subsystem 2010 includes one or more memory devices. Examples of storage devices include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of storage devices include solid-state memory and hard disk drives. In some examples, the memory is used to store computer-readable, computer-executable software, including instructions that cause the processor to perform the various functions described herein when executed. In some cases, the memory includes a basic input / output system (BIOS), which controls basic hardware or software operations, such as interaction with peripheral components or devices. In some cases, a memory controller operates the memory cells. For example, the memory controller may include a row decoder, a column decoder, or both. In some cases, the storage cells within the memory store information in the form of logical states. The memory subsystem 2010 is a reference to Figure 7 Examples of memory cells are described or include aspects thereof.

[0222] According to some embodiments, the communication interface 2015 operates at the boundary between the communication entities (such as the computing device 2000, one or more user devices, the cloud, and one or more databases) and the channel 2030 and can record and process communications. In some cases, the communication interface 2015 is provided to enable the processing system to couple to a transceiver (e.g., a transmitter and / or a receiver). In some examples, the transceiver is configured to transmit (or send) and receive signals for the communication device via an antenna. In some cases, a bus is used for the communication interface 2015.

[0223] According to some embodiments, I / O interface 2020 is controlled by an I / O controller to manage input and output signals of computing device 2000. In some cases, I / O interface 2020 manages peripheral devices that are not integrated into computing device 2000. In some cases, I / O interface 2020 represents a physical connection or port to an external peripheral device. In some cases, an I / O controller uses a computer such as or other known operating systems. In some cases, an I / O controller represents or interacts with a modem, keyboard, mouse, touch screen, or similar device. In some cases, an I / O controller is implemented as a component of a processor. In some cases, a user interacts with a device through an I / O interface 2020 or a hardware component controlled by an I / O controller. I / O interface 2020 is a reference to a device. Figure 7 Examples of I / O modules are described, or include aspects thereof.

[0224] According to some embodiments, user interface component 2025 enables a user to interact with computing device 2000. In some cases, user interface component 2025 includes an audio device, such as an external speaker system, an external display device, such as a display screen, an input device (e.g., a remote control device connected to the user interface directly or through an I / O controller), or a combination thereof.

[0225] The performance of the apparatus, system, and method of the present disclosure has been evaluated, and the results show that the embodiments of the present disclosure achieve higher performance than conventional techniques (e.g., conventional image generation systems). Example experiments demonstrate that the image processing apparatus based on the present disclosure outperforms conventional image generation systems. Reference Figures 3 to 5 Details of example use cases based on embodiments of the present disclosure are described.

[0226] The descriptions and drawings described herein represent example configurations and do not represent all implementations within the scope of the claims. For example, operations and steps may be rearranged, combined, or otherwise modified. In addition, structures and devices may be represented in block diagram form to illustrate the relationships between components and avoid obscuring the concepts being described. Similar components or features may have the same name but different reference numbers corresponding to different figures.

[0227] Some modifications to the present disclosure may be clear to those skilled in the art, and the principles defined herein may be applied to other variations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but should be given the broadest scope consistent with the principles and novel features disclosed herein.

[0228] The described methods can be implemented or performed by a device including a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor can be a microprocessor, a conventional processor, a controller, a microcontroller, or a state machine. The processor can also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration). Therefore, the functions described herein can be implemented in hardware or software and can be performed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions can be stored on a computer-readable medium in the form of instructions or code.

[0229] Computer-readable media include non-transient computer storage media and communication media, including any media that facilitate code or data transmission. Non-transient storage media can be any available media that a computer can access. For example, non-transient computer-readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc (CD) or other optical disc storage, magnetic disk storage or any other non-transient medium for carrying or storing data or code.

[0230] In addition, the connecting component can be appropriately referred to as a computer-readable medium. For example, if the code or data is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, radio, or microwave signals, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology is also included in the definition of medium. Combinations of media are also included within the scope of computer-readable media.

[0231] In this disclosure and the claims that follow, the word "or" is intended to include a list, for example, a list of X, Y, or Z means X, Y, Z, XY, XZ, YZ, or XYZ. Furthermore, the word "based on" is not intended to indicate a closed set of conditions. For example, a step described as "based on condition A" may be based on both condition A and condition B. In other words, the phrase "based on" should be interpreted as meaning "based at least in part on." Furthermore, the words "a" or "an" mean "at least one."

Claims

1. A method comprising: obtaining a structure input indicating a target spatial structure; encoding the structure input using a conditional encoder to obtain a structure encoding representing the target spatial structure; as well as A synthetic image is generated based on the structure encoding using an image generation model, wherein the synthetic image depicts an object having the target spatial structure.

2. The method of claim 1 , wherein encoding the structure input comprises: encoding each of a plurality of components of the structure input to obtain a plurality of component structure encodings, wherein each of the plurality of components comprises a different representation of the target spatial structure; combining the plurality of component structure codes to obtain a preliminary structure code; as well as The preliminary structural code is encoded using the conditional encoder to obtain the structural code.

3. The method according to claim 2, wherein: Each component structure code in the plurality of component structure codes is generated by a different structure encoder.

4. The method according to claim 2, wherein: Each of the plurality of component structure codes has a different number of channels.

5. The method according to claim 2, wherein: The plurality of component structure codes include depth code, edge code and entity code.

6. The method according to claim 1, further comprising: providing the structural encoding to a first layer of the image generation model; downsampling the structure code to obtain a downsampled structure code; as well as The downsampled structure encoding is provided to a second layer of the image generation model.

7. The method of claim 1 , wherein generating the composite image comprises: Get noise input; as well as The noisy input is denoised based on the structural encoding.

8. The method according to claim 1, further comprising: A multiple convolution process is performed on the structure code.

9. The method according to claim 1, further comprising: Structural adhesion parameters are obtained, wherein the composite image is generated based on the structural adhesion parameters using the structural encoding.

10. The method according to claim 1, further comprising: A textual cue describing the object is obtained, wherein the composite image is generated based on the textual cue.

11. The method according to claim 1 , further comprising: A style hint is obtained that indicates a style element, wherein the composite image is generated based on the style hint to include the style element.

12. The method of claim 1 , wherein obtaining the structural input comprises: Acquire preliminary images; as well as The structure input is generated based on the preliminary image.

13. The method according to claim 1, wherein The image generation model is trained using a training set comprising a training structure input indicating a spatial structure and a ground truth image comprising the target spatial structure.

14. A method for training a machine learning model, the method comprising: Acquire a training set, the training set comprising a training structure input indicating a spatial structure and a reference truth image comprising the spatial structure; as well as An image generation model is trained using the training set to generate a synthetic image based on a structure input, wherein the synthetic image includes the spatial structure.

15. The method of claim 14, wherein training the machine learning model comprises: The conditional encoder and the image generation model are jointly trained.

16. The method of claim 14, wherein training the image generation model comprises: generating a structural encoding based on the structural input; generating a predicted image based on the structure encoding; calculating a loss function by comparing the predicted image with the reference truth image; and Parameters of the image generation model are updated based on the loss function.

17. A system comprising: Memory components; a processing device coupled to the memory component, the processing device configured to perform operations comprising: obtaining a structure input indicating a target spatial structure; encoding the structure input using a conditional encoder to obtain a structure encoding representing the target spatial structure; and A synthetic image is generated based on the structure encoding using an image generation model, wherein the synthetic image depicts an object having the target spatial structure.

18. The system of claim 17, wherein: The conditional encoder includes multiple convolutional layers and multiple activation layers.

19. The system of claim 17, wherein: The image generation model includes more parameters than the conditional encoder.

20. The system of claim 17, further comprising: A text encoder configured to encode text prompts to obtain text embeddings.

Images