Generating high resolution images using diffusion models

Abstract

Embodiments of the present disclosure relate to generating high resolution images using diffusion models. Methods, non-transitory computer-readable media, apparatuses, and systems for high resolution image generation using a diffusion model include obtaining a cue and generating a predicted de-noised image of a first resolution based on the cue using a first diffusion model. A predicted de-noised image is generated at a first intermediate diffusion step of the first diffusion model. And performing up-sampling on the predicted de-noised image to obtain an up-sampled de-noised image with a second resolution, wherein the second resolution is higher than the first resolution. The second diffusion model then generates an output image of a second resolution based on the hint and the upsampled de-noised image.

Description

Technical Field

[0001] The following relates generally to machine learning, and more specifically to generating high-resolution images using diffusion models. Machine learning is a field of information processing in which algorithms or models, such as artificial neural networks, are trained to make predicted outputs in response to input data without being specifically programmed. For example, a machine learning model can be trained to predict an output image using image training data. Background Art

[0002] Diffusion models are a type of machine learning model that is well suited for image generation. In some cases, a diffusion model works by gradually removing Gaussian-distributed noise from a noisy image in a series of diffusion steps based on the diffusion model's prediction of what the denoised image should look like. As the series of diffusion steps progresses, the diffusion model's prediction of the denoised image becomes increasingly accurate. Summary of the invention

[0003] Aspects of the present disclosure provide methods, non-transitory computer-readable media, devices, and systems for generating high-resolution images using diffusion models. According to one aspect of the present disclosure, a first diffusion model generates a low-resolution noise-free prediction during an intermediate step of a back-diffusion process. In some cases, the noise-free prediction is upsampled to a higher resolution, and a second diffusion model generates a higher-resolution output image based on the upsampled noise-free prediction.

[0004] By first generating a low resolution noise-free prediction and then generating a higher resolution output based on the noise-free prediction, aspects of the present disclosure provide reduced overall processing time compared to conventional methods that use a diffusion model to generate a higher resolution output image, which would take a relatively long time to process given a similar number of diffusion steps and would produce an output image of lower quality given a significantly smaller number of diffusion steps.

[0005] In addition, some traditional image generation systems first use a diffusion model to generate a low-resolution output image, and then use various upsampling techniques to achieve a higher-resolution output image. In contrast, natively generating a higher-resolution output image based on the upsampled noise-free prediction using a second diffusion model provides a better quality and more accurate output image.

[0006] In addition, some conventional image generation systems generate a noise prediction image at an intermediate diffusion step using a first comparative diffusion model, upsample the noise prediction image, add noise to the upsampled noise image to correct the noise distribution within the upsampled noise image, and then generate a high-resolution image based on the corrected upsampled noise image using a second comparative diffusion model.

[0007] In contrast, in some cases, by upsampling the noise-free prediction of the first diffusion model at the intermediate diffusion step, aspects of the present disclosure avoid having to correct the noise distribution within the upsampled noise-free prediction because there is no noise to correct. Since the second diffusion model generates an output image based on the uncorrected upsampled noise-free prediction, the quality of the output image is higher than an output image generated based on a corrected noisy image that includes defects introduced by the noise correction process.

[0008] A method, apparatus, non-transitory computer-readable medium, and system for image generation are described. One or more aspects of the method, apparatus, non-transitory computer-readable medium, and system include: obtaining a hint; generating a predicted denoised image of a first resolution based on the hint using a first diffusion model, wherein the predicted denoised image is generated at a first intermediate diffusion step of the first diffusion model; upsampling the predicted denoised image to obtain an upsampled denoised image of a second resolution, the second resolution being higher than the first resolution; and generating an output image of a second resolution based on the hint and the upsampled denoised image using a second diffusion model.

[0009] A method, apparatus, non-transitory computer-readable medium, and system for image generation are described. One or more aspects of the method, apparatus, non-transitory computer-readable medium, and system include: using a first diffusion model to generate a predicted denoised image of a first resolution, wherein the predicted denoised image is generated at a first intermediate diffusion step of the first diffusion model; upsampling the predicted denoised image to obtain an upsampled denoised image of a second resolution, the second resolution being higher than the first resolution; performing a forward diffusion process on the upsampled denoised image to obtain an intermediate input image corresponding to a second intermediate diffusion step of the second diffusion model; and generating an output image of a second resolution based on the intermediate input image using the second diffusion model.

[0010] An apparatus, system, and method for image generation are described. One or more aspects of the apparatus, system, and method include one or more processors; one or more memory components, one or more memory components coupled to the one or more processors; a first diffusion model including first parameters stored in the one or more memory components, the first diffusion model is trained to generate a predicted denoised image of a first resolution, wherein the predicted denoised image is generated at a first intermediate diffusion step of the first diffusion model; an upsampling component configured to upsample the predicted denoised image to obtain an upsampled denoised image of a second resolution, the second resolution being higher than the first resolution; and a second diffusion model including second parameters stored in the one or more memory components, the second diffusion model is trained to generate an output image of a second resolution based on the upsampled denoised image. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 An example of an image generation system according to aspects of the present disclosure is shown.

[0012] Figure 2 An example of a method for high-resolution image generation according to aspects of the present disclosure is shown.

[0013] Figure 3 An example of a comparison image generated based on an upsampled noisy image according to aspects of the present disclosure is shown.

[0014] Figure 4 Examples of images generated based on upsampled noise-free images according to aspects of the present disclosure are shown.

[0015] Figure 5 An example of an image generating device according to aspects of the present disclosure is shown.

[0016] Figure 6 An example of a guided diffusion architecture according to aspects of the present disclosure is shown.

[0017] Figure 7 An example of a U-Net according to aspects of the present disclosure is shown.

[0018] Figure 8 An example of data flow in an image generation system according to aspects of the present disclosure is shown.

[0019] Fig. 9 An example of a method for generating a high-resolution image according to aspects of the present disclosure is shown.

[0020] Fig.10 An example of a diffusion process according to aspects of the present disclosure is shown.

[0021] Fig.11 An example of a method of generating a high-resolution image using an intermediate input image according to aspects of the present disclosure is shown.

[0022] Fig.12 An example of training a diffusion model according to aspects of the present disclosure is shown.

[0023] Fig.13 An example of a computing device according to aspects of the present disclosure is shown. DETAILED DESCRIPTION

[0024] A diffusion model is a machine learning model suitable for image generation. In some cases, a diffusion model works by gradually removing Gaussian distributed noise from a noisy image in a series of diffusion steps based on the diffusion model's prediction of what the denoised image should look like. As the series of diffusion steps progresses, the diffusion model's prediction of the denoised image becomes increasingly accurate.

[0025] Diffusion models are able to produce higher quality images than other machine learning models. However, diffusion models are trained to generate images of a specific resolution, and as the resolution of the generated images increases, the processing time and memory cost of the diffusion model also increases because more information must be processed. Therefore, generating images using diffusion models may be impractically slow and costly at higher resolutions, and the size of diffusion models may be correspondingly limited to fit in the memory of available hardware and generate images within a specified time frame.

[0026] The traditional approach to achieving high-resolution images using diffusion models involves generating low-resolution images using the diffusion model and then upsampling the low-resolution image to a higher resolution. However, this approach produces results that are not as accurate as using the diffusion model to natively generate the higher-resolution image.

[0027] Another conventional method includes using a first comparative diffusion model to generate a low-resolution noisy intermediate image at an intermediate diffusion step, resizing the noisy intermediate image to a higher resolution, and using a second comparative diffusion model to generate a higher-resolution output image based on the upsampled noisy intermediate image. This method reduces the overall processing time due to the initial generation at a lower resolution.

[0028] However, simply resizing the noisy intermediate image from a low resolution to a higher resolution results in the noise in the image being non-isotropic (e.g., non-Gaussian), and the upsampled noisy image has an uncorrected signal-to-noise ratio, which means that the second comparative diffusion model cannot make accurate noise removal predictions based on the original upsampled noisy intermediate image.

[0029] Therefore, the conventional method also includes the step of adding specific noise to the upsampled noisy intermediate image so that the noise is isotropic and Gaussian again. However, adding noise to the upsampled noisy intermediate image introduces errors into the back-diffusion process used by the second comparison diffusion model, thereby reducing the accuracy of the higher resolution output image. In addition, this conventional method is performed in pixel space using assumptions that may not apply to the latent space.

[0030] Various aspects of the present disclosure provide methods, non-transitory computer-readable media, devices, and systems for generating high-resolution images using diffusion models. According to one aspect of the present disclosure, a first diffusion model generates a low-resolution noise-free prediction during an intermediate step of a back-diffusion process. In some cases, the noise-free prediction is upsampled to a higher resolution, and a second diffusion model generates an output image at a higher resolution based on the upsampled noise-free prediction.

[0031] By first generating a low-resolution noise-free prediction and then generating a higher-resolution output image based on the noise-free prediction, aspects of the present disclosure provide reduced overall processing time and memory costs compared to conventional methods that use one diffusion model to generate a higher-resolution output image of comparable quality. Thus, aspects of the present disclosure can use larger diffusion models than conventional image generation systems can use.

[0032] Furthermore, according to some aspects, an output image generated based on the noise-free prediction more accurately depicts the intended content of the output image than a comparative output image generated by upsampling a low-resolution image to a high resolution.

[0033] In some cases, during each diffusion step of the back-diffusion process, the diffusion model predicts a partial denoised image based on the noisy input image, where the partial denoised image can be considered a combination of the diffusion step's prediction of a fully denoised image (e.g., output image) and the noise of that diffusion step. In other words, in some cases, at each diffusion step, a prediction of the denoised output image is made, and as the back-diffusion process progresses, the prediction of the denoised output image becomes increasingly accurate. Thus, in some cases, the noise-free prediction generated at an intermediate diffusion step can be considered an "incomplete" prediction of the output image, the output image generated at the last step of the back-diffusion process can be considered a "complete" prediction of the output image, and thus the second diffusion model can be considered to "complete" the noise-free prediction by generating the output image based on the noise-free prediction.

[0034] According to some aspects, since the noise-free prediction does not include noise, in contrast to the conventional methods described above, no noise correction step is required after upsampling the noise-free prediction. Therefore, since the noise correction step is avoided, the error introduced by the noise correction step is avoided, and the output image is therefore more accurate than the comparison output image. In addition, according to some aspects, both the first diffusion model and the second diffusion model are used in pixel space or latent space.

[0035] One aspect of the present disclosure is used in the context of image generation. In one example, a user instructs an image generation system to generate an image of a dog at a resolution of 128x128 pixels using the prompt "A happy dog ​​outside".

[0036] The image generation system generates a prediction x of the partially denoised image at a resolution of 64x64 pixels based on the hint at the intermediate diffusion step t of the back-diffusion process. T , where the partially denoised image consists of the noise-free prediction x0 of the output image and the noisy prediction at the intermediate diffusion step t. The image generation system then upsamples the noise-free prediction x0 to 128x128 resolution and adds noise using a forward diffusion process, returning to x at 128x128 resolution. T as an intermediate input image. The image generation system then "completes" the noise-free prediction by generating an output image based on the intermediate input image and the hints at the last step of the second back-diffusion process.

[0037] Since the first diffusion model processes approximately four times faster at 64x64 pixels than the second diffusion model at 128x128 pixels, generating noise-free prediction and output images with an equal number of diffusion steps approximately halves the total processing time compared to generating output images at only 128x128 pixel resolution.

[0038] refer to Figure 1-Figure 4 Further example applications of the present disclosure in the context of image generation are provided. Figure 1 , Figure 5-Figure 8 and Fig.13 Provides details about the architecture of the image generation system. Figure 2 and Figure 9-11 An example of the process used for image generation is provided. Fig.12 An example of the process for training an image generation model is provided.

[0039] According to some aspects of the present disclosure, methods, non-transitory computer-readable media, systems, and apparatus for high-resolution image generation using machine learning models are provided, which generate high-resolution output images faster and with less memory cost than using a traditional diffusion model to generate images of comparable quality at the same resolution, and produce more accurate output than upsampling a low-resolution output image to a high resolution or upsampling an intermediate noisy output of a first comparative diffusion model (which is then completed at a high resolution by a second comparative diffusion model).

[0040] Image generation system

[0041] refer to Figure 1-Figure 8 and Fig.13 A system and apparatus for image generation are described. One or more aspects of the system and apparatus include one or more processors; one or more memory components coupled to the one or more processors; a first diffusion model including first parameters stored in the one or more memory components, the first diffusion model trained to generate a predicted denoised image at a first resolution, wherein the predicted denoised image is generated in a first intermediate diffusion step of the first diffusion model; an upsampling component configured to upsample the predicted denoised image to obtain an upsampled denoised image at a second resolution, the second resolution being higher than the first resolution; and a second diffusion model including second parameters stored in the one or more memory components, the second diffusion model trained to generate an output image at a second resolution based on the upsampled denoised image.

[0042] Some examples of the system and the apparatus also include a forward diffusion component configured to perform a forward diffusion process on the upsampled denoised image to obtain an intermediate input image corresponding to a second intermediate diffusion step of the second diffusion model, wherein the output image is generated based on the intermediate input image. In some aspects, the second diffusion model is also trained to perform a reverse diffusion process starting from the second intermediate diffusion step.

[0043] Some examples of the system and apparatus also include a training component configured to train the first diffusion model and the second diffusion model. In some aspects, the training component is further configured to train the second diffusion model independently of the first diffusion model.

[0044] Figure 1 An example of an image generation system 100 according to aspects of the present disclosure is shown. The image generation system 100 is a reference Figure 8 Examples of corresponding elements described herein may include aspects thereof. The example shown includes a user 105, a user device 110, an image generating device 115, a cloud 120, and a database 125. The image generating device 115 is a reference Figure 5 Examples of corresponding elements are described or include aspects thereof.

[0045] See also Figure 1 According to some aspects, user 105 provides a hint (e.g., "a happy dog ​​outside") to image generation device 115 via user device 110. In some cases, image generation device 115 encodes the hint to obtain a hint embedding. In some cases, image generation device 115 generates a predicted denoised image at a first resolution using a first diffusion model and based on the hint embedding. In some cases, the predicted denoised image is generated at a first intermediate diffusion step of the first diffusion model.

[0046] In some cases, the image generation device 115 upsamples the predicted denoised image to obtain an upsampled denoised image of a second resolution, the second resolution being higher than the first resolution. In some cases, the image generation device 115 performs a forward diffusion process on the upsampled denoised image using a forward diffusion component to obtain an intermediate input image corresponding to a second intermediate diffusion step of a second diffusion model, wherein the upsampled denoised image includes noise added by the forward diffusion process.

[0047] In some cases, image generation device 115 generates an output image of a second resolution based on the intermediate input image using the second diffusion model and based on the prompt embedding. In some cases, the output image depicts the content specified by the prompt. In some cases, image generation device 115 provides the output image to user 105 via user device 110.

[0048] As used herein, "prompt" refers to information used to inform a machine learning model of the expected output, so that the output depicts the content described by the prompt. In some cases, the prompt includes information in text, images, or other modalities (such as audio) that can describe the content of the output.

[0049] As used herein, "embedding" refers to a mathematical representation of an input in a low-dimensional space that makes it easier for a machine learning model to capture and analyze information about the input. For example, in some cases, an embedding is a numerical representation of an input in a continuous vector space, where objects with similar semantic information correspond to vectors that are numerically similar and therefore "closer" to each other, thereby allowing a machine learning model to effectively compare different objects corresponding to different embeddings to each other.

[0050] As used herein, a "predicted denoised image" refers to the diffusion model's prediction of the final output x. As used herein, an "intermediate diffusion step" refers to a diffusion step that is greater than 0 but less than the maximum number of steps of the diffusion model used to perform the intermediate diffusion step, or a diffusion step that is less than the starting point of the reverse diffusion process, which is also the end point of the forward diffusion process.

[0051] According to some aspects, user device 110 is a personal computer, a laptop, a mainframe computer, a PDA, a personal assistant, a mobile device, or any other suitable processing device. In some examples, user device 110 includes software that displays a user interface (e.g., a graphical user interface) provided by image generation device 115. In some aspects, the user interface allows information (e.g., images, prompts, user input, etc.) to be transmitted between user 105 and image generation device 115.

[0052] According to some aspects, the user device user interface enables user 105 to interact with user device 110. In some embodiments, the user device user interface may include an audio device (such as an external speaker system), an external display device (such as a display screen), or an input device (e.g., a remote control device that interfaces with the user interface directly or through an I / O controller module). In some cases, the user device user interface may be a graphical user interface.

[0053] The image generation device 115 is a reference Figure 5 and Fig.13 Examples of corresponding elements described herein or including aspects thereof. According to some aspects, the image generation device 115 includes a computer-implemented network. In some embodiments, the computer-implemented network includes a machine learning model (such as, for example, a reference Figure 5-Figure 6 and Figure 8 In some embodiments, the image generation device 115 further includes one or more processors, a memory subsystem, a communication interface, an I / O interface, one or more user interface components, and a bus, as described in reference Fig.13 Furthermore, in some embodiments, the image generation device 115 communicates with the user device 110 and the database 125 via the cloud 120 .

[0054] In some cases, the image generation device 115 is implemented on a server. The server provides one or more functions to the linked user through one or more of various networks (such as, cloud 120). In some cases, the server includes a single microprocessor board, which includes a microprocessor responsible for controlling all aspects of the server. In some cases, the server uses a microprocessor and a protocol to exchange data with other devices or users on one or more networks via the hypertext transfer protocol (HTTP) and the simple mail transfer protocol (SMTP), but other protocols such as the file transfer protocol (FTP) and the simple network management protocol (SNMP) may also be used. In some cases, the server is configured to send and receive files in the hypertext markup language (HTML) format (e.g., for displaying web pages). In various embodiments, the server includes a general-purpose computing device, a personal computer, a notebook computer, a mainframe computer, a supercomputer, or any other suitable processing device.

[0055] refer to Figure 5-Figure 8 and Fig.13 More details on the architecture of the image generation device 115 are provided in reference. Figure 2-Figure 4 and Figure 9-11 More details on the image generation process are provided in the reference Fig.12 An example of the process used to train a diffusion model is provided.

[0056] Cloud 120 is a computer network configured to provide on-demand availability of computer system resources, such as data storage and computing power. In some examples, cloud 120 provides resources without active management by the user. The term "cloud" is sometimes used to describe data centers available to many users via the Internet.

[0057] Some large cloud networks have functions distributed from a central server in multiple locations. If a server has a direct or close connection with a user, the server is designated as an edge server. In some cases, the cloud 120 is limited to a single organization. In other examples, the cloud 120 is available to many organizations.

[0058] In one example, cloud 120 includes a multi-layer communication network including multiple edge routers and core routers. In another example, cloud 120 is based on a set of local switches in a single physical location. According to some aspects, cloud 120 provides communication between user device 110, image generation device 115, and database 125.

[0059] Database 125 is an organized collection of data. In one example, database 125 stores data in a specified format known as a schema. According to some aspects, database 125 is constructed as a single database, a distributed database, a plurality of distributed databases, or an emergency backup database. In some cases, a database controller manages the storage and processing of data in database 125. In some cases, a user interacts with the database controller. In other cases, the database controller operates automatically without interaction from the user. According to some aspects, database 125 is external to image generation device 115 and communicates with image generation device 115 via cloud 120. According to some aspects, database 125 is included in image generation device 115.

[0060] Figure 2 An example of a method 200 for high resolution image generation according to aspects of the present disclosure is shown. In some examples, the operations are performed by a system including a processor that executes a set of codes to control functional elements of the device. Additionally or alternatively, certain processes are performed using dedicated hardware. Typically, the operations are performed according to the methods and processes described in various aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in conjunction with other operations.

[0061] See also Figure 2 According to some aspects, an image generation system (such as reference Figure 1 and 8The described image generation system generates a high-resolution image based on a hint in the following manner: generating a noise-free image at an intermediate diffusion step of a back-diffusion process implemented by a first diffusion model, upsampling the noise-free image to a higher resolution, and generating a higher-resolution output image based on the upsampled noise-free image.

[0062] By first generating a noise-free image of a lower resolution and then generating an output image based on the noise-free image, the image generation system is able to generate an output image in a shorter time than the time required by a conventional diffusion model to generate an image of equal quality at the same resolution. In addition, because the higher resolution output image is generated using the second diffusion model, the quality of the output image is higher than the quality provided by generating the output image of the lower resolution and then upsampling the output image to the higher resolution. Finally, by upsampling the noise-free image generated at the intermediate diffusion step of the back diffusion process rather than the noise-corrected noisy image generated at the intermediate diffusion step, the image generation system avoids the degradation of image quality associated with the noise correction process.

[0063] At operation 205, the system provides a prompt. In some cases, the operation of this step refers to or can be referred to as Figure 1 For example, in some cases, the user performs the image generation through an image generating device (such as reference Figure 1 The image generating device described in the present invention) is in a user device (such as a reference Figure 1 A user interface provided on a user device (described in the embodiment of the present invention) provides a prompt to the image generating device. In some cases, the prompt is a text prompt describing the expected content of the output image.

[0064] At operation 210, the system generates a low-resolution noise-free image based on the hint using the first diffusion model at the intermediate diffusion step. In some cases, the operation of this step refers to or can be referred to as Figure 1 and Figure 5 For example, in some cases, the image generating device generates a low-resolution noise-free image, such as the reference Fig. 9 described.

[0065] At operation 215, the system upsamples the low-resolution noise-free image. In some cases, the operation of this step refers to or can be referred to as Figure 1 and Figure 5 For example, in some cases, the image generation device upsamples the low-resolution noise-free image, such as in reference Fig. 9 described.

[0066] At operation 220, the system generates a high-resolution image based on the upsampled image using the second diffusion model. In some cases, the operation of this step refers to or can be referred to as Figure 1 and Figure 5 For example, in some cases, the image generating device generates a high-resolution image, such as the reference Fig. 9 described.

[0067] At operation 225, the system provides the user with a high-resolution image. In some cases, the operation of this step is referred to or can be referred to as Figure 1 and Figure 5 The described image generation apparatus performs. For example, in some cases, the image generation apparatus provides a high-resolution image to a user via a user interface provided on a user device.

[0068] Figure 3 An example of a comparison image 300 generated based on an upsampled noisy image according to aspects of the present disclosure is shown. Figure 3 , comparison image 300 is an example of a high resolution image generated by the comparison diffusion model based on an uncorrected noise image generated at a lower resolution. Because the noise image is upsampled, the noise in the noise image is no longer isotropic and does not follow a Gaussian distribution, and therefore the signal-to-noise ratio in the upsampled image is not realistic. Because the signal-to-noise ratio is not realistic, the comparison diffusion model cannot generate comparison image 300 so that comparison image 300 is an accurate depiction of its intended content (dog).

[0069] Figure 4 An example of an image 400 generated based on an upsampled noise-free image according to aspects of the present disclosure is shown. Figure 4 , image 400 is based on a first diffusion model (such as reference Figure 5 and Figure 8 The first diffusion model described in Figure 5 and Figure 8 The upsampled noise-free prediction provided by the upsampling component described in Figure 5 and Figure 8 An example of an output image generated using the second diffusion model described in the previous section. Figure 3 Compared to the described images, image 400 includes an accurate and high quality representation of its intended content.

[0070] Figure 5 An example of an image generating device 500 according to aspects of the present disclosure is shown. The image generating device 500 is a reference Figure 1Examples of corresponding elements described or include aspects thereof. In one aspect, the image generation device 500 includes a processor unit 505 , a memory unit 510 , a first diffusion model 515 , an upsampling component 520 , a second diffusion model 525 , a forward diffusion component 530 , an encoder 535 , and a training component 540 .

[0071] The processor unit 505 includes one or more processors. A processor is an intelligent hardware device, such as a general-purpose processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof.

[0072] In some cases, processor unit 505 is configured to operate a memory array using a memory controller. In other cases, the memory controller is integrated into processor unit 505. In some cases, processor unit 505 is configured to execute computer-readable instructions stored in memory unit 510 to perform various functions. In some aspects, processor unit 505 includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing. According to some aspects, processor unit 505 includes reference Fig.13 Describes one or more processors.

[0073] Memory unit 510 includes one or more memory devices. Examples of memory devices include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of memory devices include solid-state memory and hard disk drives. In some examples, the memory is used to store computer-readable, computer-executable software including instructions that, when executed, cause at least one processor of processor unit 505 to perform various functions described herein.

[0074] In some cases, memory unit 510 includes a basic input / output system (BIOS) that controls basic hardware or software operations, such as interaction with peripheral components or devices. In some cases, memory unit 510 includes a memory controller that operates the memory units of memory unit 510. For example, the memory controller may include a row decoder, a column decoder, or both. In some cases, the memory units within memory unit 510 store information in the form of logical states. According to some aspects, memory unit 510 includes a reference Fig.13 Describes the memory subsystem.

[0075] According to some aspects, image generation device 500 uses one or more processors of processor unit 505 to execute instructions stored in memory unit 510 to perform the functions described herein. For example, in some cases, image generation device 500 obtains a prompt. In some cases, the prompt includes a text prompt.

[0076] The first diffusion model 515 is a reference Figure 8 Examples of corresponding elements described herein or include aspects thereof. According to some aspects, the first diffusion model 515 is implemented as software, firmware, one or more hardware circuits, or a combination thereof stored in the memory unit 510 and executable by the processor unit 505. According to some aspects, the first diffusion model 515 includes first parameters (e.g., machine learning parameters) stored in the memory unit 510.

[0077] Machine learning parameters, also called model parameters or weights, are variables that provide the behavior and characteristics of a machine learning model. Machine learning parameters can be learned or estimated from training data and are used to make predictions or perform tasks based on the patterns and relationships learned in the data.

[0078] Machine learning parameters are typically adjusted during the training process to minimize a loss function or maximize a performance metric. The goal of the training process is to find the best values ​​for the parameters that enable the machine learning model to make accurate predictions or perform well on a given task.

[0079] For example, during the training process, the algorithm adjusts machine learning parameters according to optimization techniques such as gradient descent, stochastic gradient descent, or other optimization algorithms to minimize the error or loss between the predicted output and the actual target. Once the machine learning parameters are learned from the training data, the machine learning parameters are used to make predictions on new, unseen data.

[0080] Artificial neural networks (ANNs) have many parameters, including weights and biases associated with each neuron in the network, which control the degree of connectivity between neurons and influence the neural network's ability to capture complex patterns in data.

[0081] An ANN is a hardware or software component that includes many connected nodes (i.e., artificial neurons) that roughly correspond to neurons in the human brain. Each connection or edge transmits a signal from one node to another (like a physical synapse in the brain). When a node receives a signal, it processes it and then transmits the processed signal to other connected nodes.

[0082] In some cases, the signals between nodes include real numbers, and the output of each node is calculated as a function of the sum of its inputs. In some examples, the nodes may use other mathematical algorithms to determine their outputs, such as selecting the maximum value from the inputs as the output, or any other suitable algorithm for activating a node. Each node and edge is associated with one or more node weights that determine how the signal is processed and transmitted.

[0083] In an ANN, a hidden layer (or intermediate layer) includes hidden nodes and is located between the input layer and the output layer. The hidden layer performs nonlinear transformations on the inputs fed into the network. Each hidden layer is trained to produce a defined output that contributes to the joint output of the output layer of the ANN. The hidden representation is a machine-readable data representation of the input learned from the hidden layers of the ANN and produced by the output layer. As the ANN is trained, the ANN's understanding of the input continues to improve, and the hidden representation gradually distinguishes itself from earlier iterations.

[0084] During the training process of an ANN, node weights are adjusted to improve the accuracy of the results (i.e., by minimizing a loss that corresponds to some degree to the difference between the current result and the target result). The weights of the edges increase or decrease the strength of the signal transmitted between nodes. In some cases, nodes have a threshold below which the signal is not transmitted. In some examples, nodes are aggregated into layers. Different layers perform different transformations on their inputs. The initial layer is called the input layer and the last layer is called the output layer. In some cases, the signal traverses certain layers multiple times.

[0085] According to some aspects, first diffusion model 515 includes one or more ANNs configured to generate a predicted denoised image at a first resolution, wherein the predicted denoised image is generated at a first intermediate diffusion step of first diffusion model 515. In some cases, first diffusion model 515 generates the predicted denoised image based on a cue.

[0086] According to some aspects, the first diffusion model 515 implements a reverse diffusion process (such as reference Figure 6 In some cases, the first diffusion model 515 includes a U-Net (such as the first back diffusion process described in reference Figure 7U-Net described in ). In some aspects, the first diffusion model 515 uses a first sampler to generate a predicted denoised image. In some cases, a "sampler" is a computational technique used to simulate samples from a probability distribution. In some cases, the diffusion model sampler simulates the evolution of a system over time, where the state of the system at each time step is sampled from a probability distribution using techniques such as Markov chain Monte Carlo (MCMC) or Sequential Monte Carlo (SMC) methods. In some cases, the sampler iteratively updates the state of the system based on observed data and prior information, thereby gradually converging to a sample of the posterior distribution.

[0087] In some cases, diffusion model samplers are particularly useful in scenarios where the dynamics of the system are not easily captured by simple parametric models, as the samplers allow incorporating complex temporal dependencies that can be exploited for tasks such as parameter estimation, uncertainty quantification, and model comparison in a Bayesian framework.

[0088] The upsampling component 520 is a reference Figure 6 and Figure 8 Examples of corresponding elements described herein or include aspects thereof. In some aspects, upsampling component 520 is implemented as software, firmware, one or more hardware circuits, or a combination thereof stored in memory unit 510 and executable by processor unit 505. In some cases, upsampling component 520 includes upsampling parameters (e.g., machine learning parameters) stored in memory unit 510.

[0089] According to some aspects, the upsampling component 520 upsamples the predicted denoised image to obtain an upsampled denoised image of a second resolution, the second resolution being higher than the first resolution. According to some aspects, the upsampling component 520 includes an image upsampling algorithm or a hardware circuit configured to perform an image upsampling algorithm, wherein the image upsampling includes increasing the spatial resolution of the image while maintaining a two-dimensional representation of the image. Examples of upsampling algorithms include bilinear upsampling, bicubic upsampling, and nearest neighbor upsampling. In some cases, the upsampling component 520 includes one or more ANNs configured to perform image upsampling.

[0090] The second diffusion model 525 is a reference Figure 8 Examples of corresponding elements described herein or include aspects thereof. According to some aspects, the second diffusion model 525 is implemented as software, firmware, one or more hardware circuits, or a combination thereof stored in the memory unit 510 and executable by the processor unit 505. According to some aspects, the second diffusion model 525 includes second parameters (e.g., machine learning parameters) stored in the memory unit 510.

[0091] According to some aspects, the second diffusion model 525 includes one or more ANNs configured to generate an output image of a second resolution based on the upsampled denoised image. In some aspects, the second diffusion model 525 generates the output image based on the hint. In some aspects, the second diffusion model 525 performs a reverse diffusion process (such as reference 50) starting from the second intermediate diffusion step. Figure 6 In some aspects, the second diffusion model 525 is trained independently of the first diffusion model 515.

[0092] In some aspects, the second diffusion model 525 generates the output image using a second sampler that is different from the first sampler. In some aspects, the second sampler includes a stochastic differential equation (SDE) solver. In some cases, an SDE solver is a computational tool or algorithm designed to numerically approximate the solution of a stochastic differential equation. A stochastic differential equation is a mathematical model used to describe how a system evolves over time when it is affected by determinism and stochasticity.

[0093] According to some aspects, the second diffusion model 525 implements a reverse diffusion process (such as reference Figure 6 In some cases, the second diffusion model 525 includes a U-Net (such as the second reverse diffusion process described in reference Figure 7 U-Net described in [5].

[0094] The forward diffusion assembly 530 is referenced Figure 8 Examples of corresponding elements described herein include aspects thereof. According to some aspects, forward diffusion component 530 is implemented as software, firmware, one or more hardware circuits, or a combination thereof stored in memory unit 510 and executable by processor unit 505 .

[0095] In some examples, the forward diffusion component 530 performs a forward diffusion process (such as reference Figure 6 5) to obtain an intermediate input image corresponding to a second intermediate diffusion step of the second diffusion model 525. In some cases, the second diffusion model 525 generates an output image based on the intermediate input image. In some examples, the forward diffusion component 530 adds noise to the upsampled denoised image, wherein the forward diffusion process is performed based on the added noise.

[0096] Encoder 535 is a reference Figure 6 and Figure 8Examples of corresponding elements described herein or include aspects thereof. According to some aspects, encoder 535 is implemented as software, firmware, one or more hardware circuits, or a combination thereof stored in memory unit 510 and executable by processor unit 505. According to some aspects, encoder 535 includes encoder parameters (e.g., machine learning parameters) stored in memory unit 510.

[0097] According to some aspects, encoder 535 includes one or more ANNs configured to encode the prompts to obtain the prompt embeddings. In some cases, first diffusion model 515, second diffusion model 525, or a combination thereof uses the prompt embeddings as input.

[0098] In some cases, encoder 535 includes a text encoder. In some cases, the text encoder includes a recurrent neural network (RNN), a transformer, or other ANN suitable for encoding text information.

[0099] A recurrent neural network (RNN) is a class of ANNs in which the connections between nodes form a directed graph along an ordered (i.e., time) sequence. This enables RNNs to model temporal dynamic behaviors, such as predicting what element should appear next in a sequence. Therefore, RNNs are suitable for tasks involving ordered sequences, such as text recognition (where words are ordered in a sentence). In some cases, RNNs include one or more finite impulse recurrent networks (characterized by nodes forming a directed acyclic graph), one or more infinite impulse recurrent networks (characterized by nodes forming a directed cyclic graph), or a combination thereof.

[0100] In some cases, the transformer includes one or more ANNs that include an attention mechanism that enables the transformer to weigh the importance of different words or tokens in the sequence. In some cases, the transformer processes the entire sequence in parallel at the same time, making the transformer efficient and allowing the transformer to capture long-range dependencies more effectively.

[0101] Attention mechanisms are a key component in some ANN architectures, especially those employed in Natural Language Processing (NLP) and sequence-to-sequence tasks, which allow the ANN to focus on different parts of the input sequence when making predictions or generating outputs.

[0102] NLP refers to the technology for interpreting or generating natural language using computers. In some cases, NLP tasks involve assigning annotation data (such as grammatical information) to words or phrases in natural language expressions. Different categories of machine learning algorithms have been applied to NLP tasks. Some algorithms (such as decision trees) use hard if-then rules. Other systems use neural networks or statistical models that make soft probabilistic decisions based on attaching real-valued weights to input features. In some cases, these models express the relative probabilities of multiple answers.

[0103] Some sequence models (such as recurrent neural networks) process the input sequence sequentially, maintaining an internal hidden state that captures information from previous steps. However, in some cases, this sequential processing makes it difficult to capture long-range dependencies or focus on specific parts of the input sequence.

[0104] The attention mechanism addresses these difficulties by enabling the ANN to selectively focus on different parts of the input sequence, assigning different degrees of importance or attention to each part. The attention mechanism achieves selective attention by considering the relevance of each input element relative to the current state of the ANN.

[0105] In some cases, an ANN using an attention mechanism receives an input sequence and maintains its current state, which represents an understanding or context. For each element in the input sequence, the attention mechanism calculates an attention score, which indicates the importance or relevance of the element given the current state. The attention score is transformed into an attention weight by a normalization process (such as applying a softmax function). The attention weight represents the contribution of each input element to the overall attention. The attention weight is used to calculate the weighted sum of the input elements to obtain a context vector. The context vector represents the information being noticed or the part of the input sequence that the ANN considers to be most relevant to the current step. The context vector is combined with the current state of the ANN to provide additional information and affect the subsequent prediction or decision of the ANN.

[0106] In some cases, by incorporating an attention mechanism, the ANN can dynamically assign attention to different parts of the input sequence, allowing the ANN to focus on relevant information and capture longer-range dependencies.

[0107] In some cases, calculating attention involves three basic steps. First, the similarity between the query vector Q obtained from the input and the key vector K is calculated to generate attention weights. In some cases, the similarity functions used for this process include dot products, splicing, detectors, etc. Next, the attention weights are normalized using a softmax function. Finally, the attention weights are weighted together with their corresponding values ​​V. In the context of an attention network, the key K and the value V are typically vectors or matrices used to represent the input data. The key K is used to determine which parts of the input the attention mechanism should focus on, while the value V is used to represent the actual data being processed.

[0108] In some cases, the transformer includes an encoder-decoder structure. In some cases, the encoder of the transformer processes an input sequence and encodes the input sequence into a set of high-dimensional representations. In some cases, the decoder of the transformer generates an output sequence based on the encoded representations and previously generated tokens. In some cases, the encoder and decoder are composed of multiple layers of self-attention mechanisms and feed-forward ANNs.

[0109] In some cases, the self-attention mechanism allows the Transformer to focus on different parts of the input sequence when computing representations for the input sequence. In some cases, the self-attention mechanism captures the relationship between the words of the sequence by assigning attention weights to each word based on its relevance to other words in the sequence, thereby enabling the Transformer to model dependencies regardless of the distance between the words.

[0110] In some cases, encoder 535 includes an image encoder trained to encode visual information, such as a convolutional neural network (CNN). CNN is a class of ANNs that are commonly used in computer vision or image classification systems. In some cases, CNN can achieve processing of digital images with minimal preprocessing. CNN can be characterized by using convolution (or cross-correlation) hidden layers. These layers apply convolution operations to the input before the result is sent to the next layer with a signal. Each convolution node can process data for a limited input field (i.e., a receptive field). During the forward pass of CNN, the filter at each layer can be convolved across the input volume to calculate the dot product between the filter and the input. During the training process, the filter can be modified so that they are activated when specific features in the input are detected.

[0111] According to some aspects, encoder 535 includes a multimodal encoder that is trained to process and represent information from multiple modalities (such as text, images, audio, or other types of data) in a multimodal embedding space. In some cases, the multimodal encoder combines information from different modalities into a unified representation that can be further used for downstream tasks such as classification, generation, or retrieval.

[0112] According to some aspects, the training component 540 is implemented as software, firmware, one or more hardware circuits, or a combination thereof stored in the memory unit 510 and executable by the processor unit 505. According to some aspects, the training component 540 is omitted from the image generation device 500. According to some aspects, the training component 500 is implemented in at least one device separate from the image generation device 500 (e.g., included in a cloud (such as a reference Figure 1 According to some aspects, a separate device (e.g., via the cloud) containing a training component 540 communicates with the image generation device 500 to perform the functions of the training component 540 described herein. According to some aspects, the training component 540 is configured to train the first diffusion model 515 and the second diffusion model 525. In some aspects, the training component 540 is also configured to train the second diffusion model 525 independently of the first diffusion model 515.

[0113] Figure 6 An example of a guided diffusion architecture 600 according to aspects of the present disclosure is shown. Diffusion models are a class of generative ANNs that can be trained to generate new data with features similar to those found in training data. In particular, diffusion models can be used to generate novel images. Diffusion models can be used for a variety of image generation tasks, including image super-resolution, generation of images with perceptual metrics, conditional generation (e.g., text-guided generation), image restoration, and image manipulation.

[0114] Diffusion models work by iteratively adding noise to the data during a forward diffusion process, and then learning to recover the data by denoising the data during a backward diffusion process. Examples of diffusion models include denoising diffusion probabilistic models (DDPMs) and denoising diffusion implicit models (DDIMs). In DDPMs, the generative process involves reversing a random Markov diffusion process. On the other hand, DDIMs use a deterministic process so that the same input produces the same output. Diffusion models can also be characterized by whether the noise is added to the image itself (as in pixel diffusion) or to image features generated by the encoder (as in latent diffusion).

[0115] For example, according to some aspects, the forward diffusion process 615 gradually adds noise to the original image 605 to obtain a noisy image 620 with various noise levels. In some cases, the forward diffusion process 615 is composed of a forward diffusion component (such as, for example, Figure 5 and Figure 8 The forward diffusion component described in FIG.

[0116] According to some aspects, the first back diffusion process 625 gradually removes noise at various noise levels from the noisy image 620 at various diffusion steps to obtain a predicted denoised image 630. In some cases, the predicted denoised image 630 is created from each of the various noise levels. For example, in some cases, at each diffusion step of the first back diffusion process 625, the first diffusion model (such as reference Figure 5 and Figure 8 The first diffusion model described above) predicts a partial denoised image, where the partial denoised image is a combination of the predicted denoised image for that diffusion step (e.g., the predicted final output) and the noise. Therefore, in some cases, each predicted denoised image can be considered as a prediction of the final noise-free output at each diffusion step by the first diffusion model, and each predicted denoised image 630 can therefore be considered as an "early" prediction of the final output at the corresponding diffusion step of the first back diffusion process 625.

[0117] According to some aspects, the predicted denoised image 630 is provided to an upsampling component 635 (such as a reference Figure 5 and Figure 8 In some cases, the upsampling component 635 upsamples the predicted denoised image 630 to output a higher resolution upsampled denoised image 640. In some cases, the forward diffusion process 615 gradually adds isotropic noise to the upsampled denoised image 640 at various noise levels to obtain an intermediate input image 645. In some cases, the intermediate input image 645 can be considered to be an enlarged version of the partial denoised image at the time step of the first back diffusion process 625 corresponding to the predicted denoised image 630, wherein the intermediate input image 645 includes a Gaussian distribution of noise.

[0118] According to some aspects, second back diffusion process 650 gradually removes noise from intermediate noisy image 645 to obtain a higher resolution output image 655. In some cases, output image 655 is created from each of a variety of noise levels.

[0119] In some cases, each of the first back-diffusion process 625 and the second back-diffusion process 650 is performed via a U-Net ANN (such as, for example, Figure 7 The forward diffusion process 615, the first reverse diffusion process 625 and the second reverse diffusion process 650 are reference Fig.10 Examples of corresponding elements are described or include aspects thereof.

[0120] In some cases, each of the first back-diffusion process 625 and the second back-diffusion process 650 is guided based on a hint 660 (such as a text hint, an image, a layout, a segmentation map, etc.). The hint 660 can be encoded using an encoder 665 (a multimodal encoder in some cases) to obtain a guidance feature 670 (e.g., a hint embedding) in a guidance space 675.

[0121] According to some aspects, the guiding features 675 are combined with the noise image 620 and the intermediate input image 645 at one or more layers of the first back diffusion process 620 and the second back diffusion process 650, respectively, to guide the predicted denoised image 630 and the output image 655 toward the content described by the inclusion hint 660. For example, the guiding features 670 can be combined with the noise image 620 and the intermediate input image 645, respectively, using a cross-attention box within the first back diffusion process 625 and the second back diffusion process 650. In some cases, the guiding features 670 can be weighted so that the guiding features 670 have a greater or lesser representation in the predicted denoised image 630 and the output image 655.

[0122] Cross-attention, also known as multi-head attention, is an extension of the attention mechanism used in some ANNs for NLP tasks. In some cases, cross-attention enables each of the first back-diffusion process 625 and the second back-diffusion process 650 to simultaneously pay attention to multiple parts of the input sequence, thereby capturing interactions and dependencies between different elements. In cross-attention, there are typically two input sequences: a query sequence and a key-value sequence. The query sequence represents the elements that need to be paid attention to, and the key-value sequence contains the elements to be noticed. In some cases, in order to calculate the cross-attention, the cross-attention block (e.g., using linear projection) transforms each element in the query sequence into a "query" representation, and transforms the elements in the key-value sequence into "key" and "value" representations.

[0123] The criss-cross attention box calculates an attention score by measuring the similarity between each query representation and the key representation, where a higher similarity indicates a higher attention given to the key element. The attention score indicates the importance or relevance of each key element to the corresponding query element.

[0124] The criss-cross attention block then normalizes the attention scores (e.g., using a softmax function) to obtain attention weights, where the attention weights determine how much information in each value element is incorporated into the final attention representation. By simultaneously attending to different parts of the key-value sequence, the criss-cross attention block captures the relationships and dependencies between the input sequences, thereby allowing each of the first back-diffusion process 625 and the second back-diffusion process 650 to better understand the context and generate more accurate and context-relevant outputs.

[0125] like Figure 6 As shown in , the guided diffusion architecture 600 is implemented according to a pixel diffusion model. According to some aspects, the guided diffusion architecture 600 is implemented according to a latent diffusion model. In the latent diffusion model, the forward and backward diffusion processes occur in a latent space instead of a pixel space.

[0126] For example, in some cases, the image encoder encodes the original image 605 as image features in the latent space. In some cases, the forward diffusion process 615 adds noise to the image features instead of the original image 605 to obtain the noisy image features. In some cases, the first reverse diffusion process 625 gradually removes noise from the noisy image features (in some cases, guided by the guide feature 670) to obtain the predicted denoised image features at the intermediate step of the first reverse diffusion process 625. In some cases, the upsampling component upsamples the predicted denoised image features to obtain the upsampled image features. In some cases, the forward diffusion process 615 gradually adds noise to the upsampled image features to obtain the intermediate image features. In some cases, the second reverse diffusion process 650 gradually removes noise from the intermediate image features to obtain the output image features.

[0127] In some cases, the image decoder decodes the output image features to obtain an output image 655 in the pixel space 610. In some cases, since the size of the image features in the latent space may be significantly smaller than the image resolution in the pixel space (e.g., 32x32, 64x64, etc. compared to 256x256, 512x512, etc.), encoding the original image 605 to obtain the image features can significantly reduce the inference time.

[0128] Figure 7 An example of a U-Net 700 according to aspects of the present disclosure is shown. According to some aspects, a diffusion model (such as reference Figure 5 and Figure 8 The first diffusion model or the second diffusion model described herein includes an ANN architecture referred to as a U-Net. In some cases, the U-Net 700 implements the reference Figure 6 , Figure 8 , Fig.10 and Fig.12 Describe the back diffusion process.

[0129] According to some aspects, U-Net 700 receives input features 705, where the input features 705 include an initial resolution and an initial number of channels, and processes the input features 705 using an initial neural network layer 710 (e.g., a convolutional neural network layer) to generate intermediate features 715.

[0130] In some cases, the intermediate features 715 are then downsampled using a downsampling layer 720 such that the downsampled features 725 have a resolution less than the initial resolution and a number of channels greater than the initial number of channels.

[0131] In some cases, the process is repeated multiple times and then the process is reversed. For example, the downsampled features 725 are upsampled using the upsampling process 730 to obtain upsampled features 735. In some cases, the upsampled features 735 are combined with the intermediate features 715 having the same resolution and number of channels via a skip connection 740. In some cases, the combination of the intermediate features 715 and the upsampled features 735 is processed using a final neural network layer 745 to produce output features 750. In some cases, the output features 750 have the same resolution as the initial resolution and the same number of channels as the initial number of channels.

[0132] According to some aspects, U-Net 700 receives additional input features to produce conditionally generated outputs. In some cases, the additional input features include vector representations of prompts. In some cases, the additional input features are combined with intermediate features 715 within U-Net 700 at one or more layers. For example, in some cases, a cross-attention module is used to combine the additional input features and the intermediate features 715.

[0133] Figure 8 An example of data flow in an image generation system 800 according to aspects of the present disclosure is shown. The example shown includes the image generation system 800, hints 830, hint embeddings 835, predicted denoised images 840, upsampled denoised images 845, intermediate input images 850, and output images 855.

[0134] Image generation system 800 is a reference Figure 1 Examples of corresponding elements described herein include aspects thereof. In one aspect, the image generation system 800 includes an encoder 805 , a first diffusion model 810 , an upsampling component 815 , a forward diffusion component 820 , and a second diffusion model 825 .

[0135] Encoder 805 is a reference Figure 5 and Figure 6 The first diffusion model 810 is a reference to the corresponding element of the example or includes aspects thereof. Figure 5 Examples of corresponding elements described herein or including aspects thereof. The upsampling component 815 is a reference Figure 5 and Figure 6 Examples of corresponding elements described herein or including aspects thereof. Forward diffusion assembly 820 is referenced Figure 5 The second diffusion model 825 is a reference to the corresponding element of the example or includes aspects thereof. Figure 5Examples of corresponding elements are described or include aspects thereof.

[0136] See also Figure 8 According to some aspects, encoder 805 receives hint 830. In some cases, encoder 805 encodes hint 830 to obtain hint embedding 835. In some cases, encoder 805 provides hint embedding 835 to first diffusion model 810. In some cases, first diffusion model 810 generates predicted denoised image 840 based on hint embedding 835.

[0137] According to some aspects, upsampling component 815 upsamples predicted denoised image 840 to a higher resolution to obtain upsampled denoised image 845. In some cases, forward diffusion component 820 adds noise to upsampled denoised image 845 to obtain intermediate input image 850.

[0138] According to some aspects, second diffusion model 825 receives as input hint embedding 835 and intermediate input image 850. In some cases, second diffusion model 825 generates output image 855 based on hint embedding 835 and intermediate input image 850.

[0139] Image Generation

[0140] refer to Figure 9-10 A method for image generation is described. One or more aspects of the method include: obtaining a cue; generating a predicted denoised image at a first resolution based on the cue using a first diffusion model, wherein the predicted denoised image is generated at a first intermediate diffusion step of the first diffusion model; upsampling the predicted denoised image to obtain an upsampled denoised image at a second resolution, the second resolution being higher than the first resolution; and generating an output image at a second resolution based on the cue and the upsampled denoised image using a second diffusion model. In some aspects, the second diffusion model is trained independently of the first diffusion model.

[0141] Some examples of the method also include performing a forward diffusion process on the upsampled denoised image to obtain an intermediate input image corresponding to a second intermediate diffusion step of a second diffusion model, wherein the output image is generated based on the intermediate input image.

[0142] Some examples of the method further include adding noise to the upsampled denoised image, wherein a forward diffusion process is performed based on the added noise. In some aspects, the second diffusion model performs a backward diffusion process starting from the second intermediate diffusion step.

[0143] Some examples of the method also include encoding the prompt to obtain a prompt embedding, wherein the first diffusion model and the second diffusion model use the prompt embedding as an input.

[0144] In some aspects, the first diffusion model uses a first sampler to generate a predicted denoised image, and the second diffusion model uses a second sampler different from the first sampler to generate an output image. In some aspects, the second sampler includes a stochastic differential equation (SDE) solver.

[0145] refer to Figure 10-11 A method for image generation is described. One or more aspects of the method include: using a first diffusion model to generate a predicted denoised image of a first resolution, wherein the predicted denoised image is generated at a first intermediate diffusion step of the first diffusion model; upsampling the predicted denoised image to obtain an upsampled denoised image of a second resolution, the second resolution being higher than the first resolution; performing a forward diffusion process on the upsampled denoised image to obtain an intermediate input image corresponding to a second intermediate diffusion step of the second diffusion model; and generating an output image of a second resolution based on the intermediate input image using the second diffusion model. In some aspects, the second diffusion model is trained independently of the first diffusion model.

[0146] Some examples of the method further include adding noise to the upsampled denoised image, wherein a forward diffusion process is performed based on the added noise. In some aspects, the second diffusion model performs a backward diffusion process starting from the second intermediate diffusion step.

[0147] Some examples of the method also include encoding the prompt to obtain a prompt embedding, wherein the first diffusion model and the second diffusion model use the prompt embedding as an input.

[0148] In some aspects, the first diffusion model uses a first sampler to generate a predicted denoised image, and the second diffusion model uses a second sampler different from the first sampler to generate an output image. In some aspects, the second sampler includes a stochastic differential equation (SDE) solver.

[0149] Fig. 9 An example of a method 900 for generating a high resolution image according to aspects of the present disclosure is shown. In some examples, the operations are performed by a system including a processor that executes a set of codes to control functional elements of the device. Additionally or alternatively, certain processes are performed using dedicated hardware. Typically, the operations are performed according to the methods and processes described in various aspects of the present disclosure. In some cases, each operation described herein is composed of various sub-steps or is performed in conjunction with other operations.

[0150] See also Fig. 9 According to some aspects of the present disclosure, a method for generating a high-resolution image using multiple diffusion models is provided. For example, in some cases, a first diffusion model generates a predicted denoised image at a first resolution at an intermediate (e.g., non-final) diffusion step of a first back diffusion process.

[0151] In some cases, at each diffusion step of the back-diffusion process, the diffusion model predicts a partial denoised image based on the noisy input image, where the partial denoised image can be considered a combination of the diffusion step's predictions of the full denoised image (e.g., output image) and the noise. In other words, in some cases, at each diffusion step, a prediction of the output image is made, and as the back-diffusion process progresses, the prediction of the output image becomes increasingly accurate. Therefore, in some cases, the predicted denoised image generated at the intermediate diffusion steps can be considered an "incomplete" prediction of the output image.

[0152] In some cases, the predicted denoised image is upsampled to a higher resolution than the first resolution to obtain an intermediate input image, and then the second diffusion model generates a higher resolution output image based on the intermediate input image.

[0153] According to some aspects, by first generating a partially denoised image of lower resolution and then generating a higher resolution output image based on the partially denoised image, an output image is provided in a shorter time than an image of comparable quality and resolution would be generated by using a conventional method of performing a back diffusion process only at a higher resolution to generate a comparative image.

[0154] Furthermore, according to some aspects, by upsampling the predicted denoised image generated at an intermediate diffusion step of the first backdiffusion process, the image generation system avoids having to correct the signal-to-noise ratio in the upsampled predicted denoised image (because the predicted denoised image does not contain noise).

[0155] In contrast, conventional methods use a comparative first diffusion model to generate a partial denoised image at an intermediate diffusion step of a comparative back-diffusion process, upsample the partial denoised image to a higher resolution (which causes the noise distribution in the upsampled partial denoised image to be non-Gaussian and therefore unsuitable for use in the back-diffusion process), add noise to the upsampled partial denoised image so that the noise distribution is Gaussian again, and use a comparative second diffusion model to generate a comparative output image based on the corrected upsampled partial denoised image. According to some aspects, since the upsampled predicted denoised image does not include noise, noise correction does not need to be performed on the upsampled predicted denoised image, and thus the output image is more accurate than the comparative output image because errors introduced by noise correction are avoided.

[0156] Finally, according to some aspects, the predicted denoised image can be considered as an "incomplete" or "preliminary" output image that partially depicts the intended content for the output image, while the output image can be considered as a complete predicted denoised image that fully depicts the intended content. Thus, in some cases, the output image generated based on the upsampled predicted denoised image includes a more accurate depiction of the intended content than the comparative output image generated by upsampling the "complete" output image of the comparative diffusion model from a lower resolution to a higher resolution.

[0157] At operation 905, the system obtains a prompt. In some cases, the operation of this step refers to or can be referred to as Figure 1 and Figure 5 For example, in some cases, a user (such as a reference Figure 1 The user described) via a user device (such as reference Figure 1 In some cases, the user provides the prompt to the image generation device via a user interface provided on the user device. In some cases, the image generation device obtains the prompt from a database (such as a reference Figure 1 Described database) retrieve prompt.

[0158] In some cases, the prompt is a text prompt that includes text. In some cases, the prompt is an image prompt that includes an image. In some cases, the prompt is a prompt that includes content in a non-text, non-image modality (such as audio). In some cases, the prompt includes information describing the expected content of the output image to be generated by the image generation device.

[0159] At operation 910, the system generates a predicted denoised image at a first resolution based on the hint using a first diffusion model, wherein the predicted denoised image is generated at a first intermediate diffusion step of the first diffusion model. In some cases, the operation of this step refers to or can be referred to as Figure 5 and Figure 8 The first diffusion model implementation described is described.

[0160] For example, in some cases, a forward diffusion component (such as, for example, Figure 5 and Figure 8 The forward diffusion assembly described in Figure 6 and Fig.10 In some cases, the first diffusion model uses a reverse diffusion process (such as the forward diffusion process described in Figure 6 The first back diffusion process described and referenced Fig.10 The back diffusion process described in ) generates the predicted denoised image.

[0161] In some cases, an encoder (such as a reference Figure 5-Figure 6 and Figure 8 The first diffusion model may be configured to encode the hint using an encoder (described in the embodiment of the present invention) to obtain a hint embedding. In some cases, the first diffusion model uses the hint embedding as an input. For example, in some cases, the first diffusion model uses the hint embedding as a guiding feature to generate a predicted denoised image. In some cases, the first diffusion model uses the first sampler to generate a predicted denoised image.

[0162] At operation 915, the system upsamples the predicted denoised image to obtain an upsampled denoised image at a second resolution (e.g., 128x128 pixels), the second resolution being higher than the first resolution (e.g., 64x64 pixels). In some cases, the operation of this step may refer to or be performed by reference to Figure 5 and Figure 8 The upsampling component described performs the above.

[0163] At operation 920, the system generates an output image at a second resolution based on the hint and the upsampled denoised image using a second diffusion model. In some cases, the operation of this step refers to or can be referred to as Figure 5 and Figure 8 The second diffusion model described is implemented.

[0164] For example, in some cases, the forward diffusion component performs a forward diffusion process on the upsampled denoised image (such as reference Figure 6 and Fig.10 In some cases, the forward diffusion model adds noise to the upsampled denoised image, wherein the forward diffusion process is based on the added noise. In some cases, a correct signal-to-noise ratio is maintained in the intermediate input image due to the addition of noise using the forward diffusion process.

[0165] In some cases, the second diffusion model performs a reverse diffusion process starting from the second intermediate diffusion step (such as reference Figure 6 The second reverse diffusion process described and referenced Fig.10 The back diffusion process described in ) is used to obtain the output image based on the intermediate input image.

[0166] In some cases, the second diffusion model uses the prompt embedding as input. For example, in some cases, the second diffusion model uses the prompt embedding as a guiding feature to generate an output image. In some cases, the output image depicts the content described by the prompt.

[0167] In some cases, the second diffusion model uses a second sampler to generate an output image. In some cases, the second sampler is different from the first sampler. In some cases, the second sampler includes a stochastic differential equation (SDE) solver. In some cases, using an SDE solver allows for more robust upsampling, where the added noise interferes with the resized image. In some cases, this reduces the impact of potential artifacts from previous steps.

[0168] In some cases, the image generating device provides the output image to the user. For example, in some cases, the image generating device displays the output image to the user via a user interface provided on the user device through the image generating device.

[0169] Fig.10 An example 1000 of a diffusion process according to aspects of the present disclosure is shown. The example shown includes a forward diffusion process 1005 (such as, for example, Figure 6 The forward diffusion process described) and the reverse diffusion process 1010 (such as, for example, Figure 6 In some cases, the forward diffusion process 1005 adds noise to the image (or image features in the latent space). In some cases, the back diffusion process 1010 denoises the image (or image features in the latent space) to obtain a denoised image.

[0170] According to some aspects, according to the known variance plan 0<β1<β2<…<β T <1, forward diffusion components (such as reference Figure 5-Figure 6 and Figure 8 The forward diffusion component described in ) uses the forward diffusion process 1005 to iteratively add Gaussian noise to the input at each diffusion step t:

[0171]

[0172] According to some aspects, Gaussian noise is obtained by Take samples and set And from the mean And the variance σ 2 =β t ≥ 1. Therefore, starting from the initial input x0, the forward diffusion process 1005 generates x1, ..., x t ,...x T , where x T is pure Gaussian noise.

[0173] In some cases, a Markov chain is used to map an observed variable x0 (such as the original image 1030) to intermediate variables x1, ..., x2 in pixel space or latent space.T , where the intermediate variables x1, …, x T has the same dimension as the observed variable x0. In some cases, the Markov chain gradually adds Gaussian noise to the observed variable x0 or to the intermediate variables x1, ..., x T , to obtain the approximate posterior q(x1:T|x0).

[0174] According to some aspects, during the reverse diffusion process 10 10, a diffusion model (such as reference Figure 5 and Figure 8 The first diffusion model or the second diffusion model described by T Remove the noise from the image 1030 to obtain a prediction for the observed variable x0 (e.g., a representation of what the diffusion model thinks the original image 1030 should be). In some cases, the prediction is guided by a guidance hint or guidance vector (e.g., a reference Figure 6 However, the diffusion model does not know the conditional distribution p(x t-1 |x t ), since computing the conditional distribution would require knowledge of the distribution of all possible images. Therefore, the diffusion model is trained to approximate (e.g., learn) the conditional distribution p(x t-1 |x t )’s conditional probability distribution p θ (x t-1 |x t ):

[0175]

[0176] In some cases, the conditional probability distribution p θ (x t-1 |x t ) is given by μ θ parameterized, and the conditional probability distribution p θ (x t-1 |x t ) is given by ∑ θ Parameterization. In some cases, the mean and variance are conditioned on a noise level t (eg, the amount of noise corresponding to diffusion step t). According to some aspects, the diffusion model is trained to learn the mean and / or variance.

[0177] According to some aspects, the diffusion model uses the noisy data x T (such as a noise image 1015) initiates the back diffusion process 1010. According to some aspects, the diffusion model iteratively applies the noise data x T Denoising is performed to obtain the conditional probability distribution p θ (x t-1 |x t). For example, in some cases, at each step t-1 of the back diffusion process 1010, the diffusion model converts x t (such as the first intermediate image 1020) and t, where t represents a step in the transformation sequence associated with different noise levels, and iteratively outputs x t-1 (such as the second intermediate image 1025), until the noise data x T is restored to be a prediction of the observed variable x0 (eg, a predicted image of the original image 1030).

[0178] According to some aspects, the joint probability of a sequence of samples in a Markov chain is determined as the product of the conditional and marginal probabilities:

[0179]

[0180] In some cases, is a pure noise distribution because the backward diffusion process 1010 transforms the result of the forward diffusion process 1005 (e.g., pure noise x T samples) as input, and Represents the Gaussian transformed sequence corresponding to a sequence with Gaussian noise added to the samples.

[0181] Fig.11 An example of a method 1100 for generating a high resolution image using an intermediate input image according to aspects of the present disclosure is shown. In some examples, these operations are performed by a system including a processor that executes a set of codes to control functional elements of the device. Additionally or alternatively, certain processes are performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described in various aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or performed together with other operations.

[0182] At operation 1105, the system generates a predicted denoised image at a first resolution using a first diffusion model, wherein the predicted denoised image is generated at a first intermediate diffusion step of the first diffusion model. In some cases, the operation of this step refers to or can be referred to as Figure 5 and Figure 8 The first diffusion model implementation described.

[0183] At operation 1110, the system upsamples the predicted denoised image to obtain an upsampled denoised image at a second resolution, the second resolution being higher than the first resolution. In some cases, the operation of this step refers to or can be referred to as Figure 5-Figure 6 and Figure 8 The upsampling component described performs the above.

[0184] At operation 1115, the system performs a forward diffusion process on the upsampled denoised image to obtain an intermediate input image corresponding to a second intermediate diffusion step of the second diffusion model. In some cases, the operation of this step refers to or can be referred to as Figure 5 and Figure 8 The forward diffusion assembly described is performed.

[0185] At operation 1120, the system generates an output image at a second resolution based on the intermediate input image using a second diffusion model. In some cases, the operation of this step refers to or can be referred to as Figure 5 and Figure 8 The second diffusion model described is implemented.

[0186] train

[0187] Fig.12 An example of a method 1200 for training a diffusion model according to aspects of the present disclosure is shown. In some examples, the operations are performed by a system including a processor that executes a set of codes to control functional elements of the device. Additionally or alternatively, certain processes are performed using dedicated hardware. Generally, the operations are performed according to the methods and processes described in various aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or performed together with other operations.

[0188] refer to Fig.12 According to some aspects, a training component (such as a reference Figure 5 training components described in the reference Figure 5 and Figure 8 The first diffusion model or the second diffusion model described above) is used to generate an image.

[0189] At operation 1205, the system initializes the diffusion model. In some cases, the operation of this step refers to or can be referred to as Figure 5 In some cases, the initialization includes defining the architecture of the diffusion model and establishing initial values ​​for the parameters of the diffusion model. In some cases, the training component initializes the diffusion model to implement a U-Net architecture (such as, for example, reference Figure 7 In some cases, initialization includes defining hyperparameters of the diffusion model architecture, such as the number of layers, the resolution and channels of each layer box, the location of skip connections, etc.

[0190] At operation 1210, the system uses a forward diffusion process (such as that described in reference to Figure 6 and Fig.10 The forward diffusion process described in ) adds noise to the training image. In some cases, the operation of this step refers to or can be referred to Figure 5In some cases, the training component retrieves data from a database (such as a reference Figure 1 The training images are retrieved from the database described in [4].

[0191] At operation 1215, at each stage n, starting from stage N, the system uses a reverse diffusion process (such as reference Figure 6 and Fig.10 In some cases, the back diffusion process described above predicts an image for stage n-1. In some cases, the operation of this step refers to or can be performed by the diffusion model. In some cases, each stage n corresponds to a diffusion step t. In some cases, at each stage n, the diffusion model predicts the noise that can be removed from the intermediate image to obtain a predicted image. In some cases, the original image is predicted at each stage of the training process.

[0192] In some cases, the back diffusion process is conditioned on a prompt. In some cases, an encoder (such as a reference Figure 5-Figure 6 and Figure 8 The encoder described in the embodiment of the present invention obtains hints and generates guiding features in the guiding space. In some cases, at each stage, the diffusion model predicts the noise that can be removed from the intermediate image to obtain a predicted image that is aligned with the guiding features.

[0193] At operation 1220, the system compares the predicted image at stage n-1 with an actual image, such as the image at stage n-1 or the original input image (eg, training image). In some cases, the operation of this step refers to or can be referred to as Figure 5 The training component described herein performs. In some cases, the training component calculates a loss function based on the comparison.

[0194] A loss function is a function that affects how a machine learning model is trained in a supervised learning model. For example, during each training iteration, the output of the machine learning model is compared to the known annotation information in the training data. The loss function provides a value (“loss”) for how close the predicted annotated data is to the actual annotated data. After calculating the loss, the model’s parameters are updated accordingly, and a new set of predictions are made during the next iteration.

[0195] Supervised learning is a machine learning technique based on learning a function that maps inputs to outputs based on example input-output pairs. Supervised learning generates a function that predicts labeled data based on labeled training data consisting of a set of training examples. In some cases, each example is a pair consisting of an input object (usually a vector) and a desired output value (i.e., a single value or output vector). In some cases, the supervised learning algorithm analyzes the training data and generates an inferred function that can be used to map new examples. In some cases, learning results in a function that correctly determines the class label of unseen instances. In other words, the learning algorithm generalizes from the training data to unseen examples.

[0196] At operation 1225, the system updates the parameters of the diffusion model based on the comparison. In some cases, the operation of this step refers to or can be referred to as Figure 5 In some cases, the training component updates machine learning parameters of the diffusion model based on the loss function. For example, in some cases, the training component updates parameters of the U-Net using gradient descent. In some cases, the training component trains the U-Net to learn time-dependent parameters of the Gaussian transformation. In some cases, the training component optimizes for negative log-likelihood.

[0197] In some cases, the training component trains the second diffusion model independently of the first diffusion model. In some cases, the training component trains the first diffusion model to generate an output image at a first resolution. In some cases, the training component trains the second diffusion model to generate an output image having a second resolution that is higher than the first resolution.

[0198] Fig.13 An example of a computing device 1300 according to aspects of the present disclosure is shown. According to some aspects, computing device 1300 includes processor(s) 1305, memory subsystem 1310, communication interface 1315, I / O interface 1320, user interface component(s) 1325, and channel 1330.

[0199] In some embodiments, computing device 1300 is a reference Figure 1 and Figure 5An example of an image generation device described or including aspects thereof. In some embodiments, the computing device 1300 includes one or more processors 1305, which can execute instructions stored in the memory subsystem 1310 to obtain a hint; generate a predicted denoised image at a first resolution based on the hint using a first diffusion model, wherein the predicted denoised image is generated at a first intermediate diffusion step of the first diffusion model; upsample the predicted denoised image to obtain an upsampled denoised image at a second resolution, the second resolution being higher than the first resolution; and generate an output image at a second resolution based on the hint and the upsampled denoised image using a second diffusion model.

[0200] According to some aspects, computing device 1300 includes one or more processors 1305. Processor(s) 1305 are reference Figure 5 Examples of processor units described herein include aspects thereof. In some cases, a processor is an intelligent hardware device, (e.g., a general purpose processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or a combination thereof.

[0201] In some cases, the processor is configured to operate the memory array using a memory controller. In other cases, the memory controller is integrated into the processor. In some cases, the processor is configured to execute computer-readable instructions stored in the memory to perform various functions. In some embodiments, the processor includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing.

[0202] According to some aspects, the memory subsystem 1310 includes one or more memory devices. The memory subsystem 1310 is a reference Figure 5 Examples of the described memory units may include aspects thereof. Examples of memory devices include random access memory (RAM), read-only memory (ROM), or hard disk. Examples of memory devices include solid-state memory and hard disk drives. In some examples, the memory is used to store computer-readable, computer-executable software including instructions that, when executed, cause the processor to perform the various functions described herein. In some cases, the memory includes, among other things, a basic input / output system (BIOS), which controls basic hardware or software operations, such as interaction with peripheral components or devices. In some cases, a memory controller operates the memory unit. For example, the memory controller may include a row decoder, a column decoder, or both. In some cases, the memory cells within the memory bank store information in the form of logical states.

[0203] According to some aspects, the communication interface 1315 operates at the boundary between the communication entities (such as the computing device 1300, one or more user devices, the cloud, and one or more databases) and the channel 1330, and can record and process the communication. In some cases, the communication interface 1315 is provided to enable the processing system to be coupled to a transceiver (e.g., a transmitter and / or a receiver). In some examples, the transceiver is configured to transmit (or send) and receive signals for the communication device via an antenna.

[0204] According to some aspects, I / O interface 1320 is controlled by an I / O controller to manage input and output signals for computing device 1300. In some cases, I / O interface 1320 manages peripheral devices that are not integrated into computing device 1300. In some cases, I / O interface 1320 represents a physical connection or port to an external peripheral device. In some cases, an I / O controller uses an operating system, such as or other known operating systems. In some cases, an I / O controller represents or interacts with a modem, keyboard, mouse, touch screen, or similar device. In some cases, an I / O controller is implemented as a component of a processor. In some cases, a user interacts with the device via an I / O interface 1320 or through a hardware component controlled by an I / O controller.

[0205] According to some aspects, user interface component(s) 1325 enable a user to interact with computing device 1300. In some cases, user interface component(s) 1325 include an audio device such as an external speaker system, an external display device such as a display screen, an input device (e.g., a remote control device that interfaces with a user interface directly or through an I / O controller), or a combination thereof. In some cases, user interface component(s) 1325 include a GUI.

[0206] The descriptions and drawings described herein represent example configurations and do not represent all implementations within the scope of the claims. For example, operations and steps may be rearranged, combined, or otherwise modified. In addition, structures and devices may be represented in the form of block diagrams to represent the relationships between components and avoid obscuring the concepts described. Similar components or features may have the same name, but may have different reference numerals corresponding to different figures.

[0207] Those skilled in the art can easily make some modifications to the present disclosure, and the principles defined herein can be applied to other variations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but should be given the broadest scope consistent with the principles and novel features disclosed herein.

[0208] The described methods may be implemented or performed by a device including a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components or any combination thereof. A general purpose processor may be a microprocessor, a conventional processor, a controller, a microcontroller or a state machine. The processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration). Therefore, the functions described herein may be implemented in hardware or software and may be performed by a processor, firmware or any combination thereof. If implemented in software executed by a processor, the functions may be stored in a computer readable medium in the form of instructions or codes.

[0209] Computer-readable media include non-transient computer storage media and communication media, including any medium that facilitates the transmission of code or data. Non-transient storage media can be any available medium that can be accessed by a computer. For example, non-transient computer-readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk (CD) or other optical disk storage, magnetic disk storage, or any other non-transient medium for carrying or storing data or code.

[0210] In addition, the connecting component may be appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared, radio, or microwave signals) is used to transmit code or data from a website, server, or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology is included in the definition of medium. Combinations of media are also included within the scope of computer-readable media.

[0211] In this disclosure and the following claims, the word "or" indicates an inclusive list, such as, for example, a list of X, Y, or Z means X or Y or Z or XY or XZ or YZ or XYZ. In addition, the phrase "based on" is not used to indicate a closed set of conditions. For example, a step described as "based on condition A" can be based on both condition A and condition B. In other words, the phrase "based on" should be interpreted as "based at least in part on". In addition, "a" or "an" indicates "at least one".

Claims

1. A method for image generation, comprising: Get tips; generating a predicted denoised image at a first resolution based on the hint using a first diffusion model, wherein the predicted denoised image is generated at a first intermediate diffusion step of the first diffusion model; Upsampling the predicted denoised image to obtain an upsampled denoised image at a second resolution, wherein the second resolution is higher than the first resolution; as well as An output image at the second resolution is generated based on the hint and the upsampled denoised image using a second diffusion model.

2. The method of claim 1, wherein generating the output image further comprises: A forward diffusion process is performed on the upsampled denoised image to obtain an intermediate input image corresponding to a second intermediate diffusion step of the second diffusion model, wherein the output image is generated based on the intermediate input image.

3. The method according to claim 2, further comprising: Noise is added to the upsampled denoised image, wherein the forward diffusion process is performed based on the added noise.

4. The method according to claim 1, wherein: The second diffusion model performs a reverse diffusion process starting from a second intermediate diffusion step.

5. The method according to claim 1, further comprising: The hint is encoded to obtain a hint embedding, wherein the first diffusion model and the second diffusion model use the hint embedding as input.

6. The method according to claim 1, wherein: The first diffusion model uses a first sampler to generate the predicted denoised image, and the second diffusion model uses a second sampler different from the first sampler to generate the output image.

7. The method according to claim 6, wherein: The second sampler includes a stochastic differential equation (SDE) solver.

8. The method according to claim 1, wherein: The second diffusion model is trained independently of the first diffusion model.

9. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to: generating a predicted denoised image at a first resolution using a first diffusion model, wherein the predicted denoised image is generated at a first intermediate diffusion step of the first diffusion model; Upsampling the predicted denoised image to obtain an upsampled denoised image at a second resolution, wherein the second resolution is higher than the first resolution; performing a forward diffusion process on the upsampled denoised image to obtain an intermediate input image corresponding to a second intermediate diffusion step of the second diffusion model; as well as An output image at the second resolution is generated based on the intermediate input image using a second diffusion model.

10. The non-transitory computer readable medium of claim 9, wherein the instructions further cause the processor to: Noise is added to the upsampled denoised image, wherein the forward diffusion process is performed based on the added noise.

11. The non-transitory computer readable medium of claim 9, wherein: The second diffusion model performs a reverse diffusion process starting from a second intermediate diffusion step.

12. The non-transitory computer readable medium of claim 9, wherein the instructions further cause the processor to: The hint is encoded to obtain a hint embedding, wherein the first diffusion model and the second diffusion model use the hint embedding as input.

13. The non-transitory computer readable medium of claim 9, wherein: The first diffusion model uses a first sampler to generate the predicted denoised image, and the second diffusion model uses a second sampler different from the first sampler to generate the output image.

14. The non-transitory computer readable medium of claim 13, wherein: The second sampler includes a stochastic differential equation (SDE) solver.

15. The non-transitory computer readable medium of claim 9, wherein: The second diffusion model is trained independently of the first diffusion model.

16. A system for image generation, wherein the system comprises: one or more processors; one or more memory components coupled to the one or more processors; a first diffusion model, the first diffusion model comprising first parameters stored in the one or more memory components, the first diffusion model being trained to generate a predicted denoised image at a first resolution, wherein the predicted denoised image is generated at a first intermediate diffusion step of the first diffusion model; an upsampling component configured to upsample the predicted denoised image to obtain an upsampled denoised image at a second resolution, the second resolution being higher than the first resolution; as well as A second diffusion model including second parameters stored in the one or more memory components is trained to generate an output image at the second resolution based on the upsampled denoised image.

17. The system of claim 16, further comprising: A forward diffusion component is configured to perform a forward diffusion process on the upsampled denoised image to obtain an intermediate input image corresponding to a second intermediate diffusion step of the second diffusion model, wherein the output image is generated based on the intermediate input image.

18. The system of claim 16, wherein: The second diffusion model is also trained to perform a reverse diffusion process starting from the second intermediate diffusion step.

19. The system of claim 16, further comprising: A training component is configured to train the first diffusion model and the second diffusion model.

20. The system of claim 19, wherein: The training component is also configured to train the second diffusion model independently of the first diffusion model.

Images