Content synthesis using generative artificial intelligence models

Abstract

A method includes receiving a cue describing a desired characteristic of an image. The method further includes generating a cue code based on the cue using the set of coding models. The method further includes generating a first cue embedding and a first image embedding based on the cue code and the noise input using a first transducer block in the diffusion transducer model. The method further includes generating a second image embedding based on the first image embedding and the first hint embedding using a second transducer block in the diffusion transducer model. The method further includes generating the image based on the second image embedding.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 567,127, filed on March 19, 2024, and U.S. Provisional Patent Application No. 63 / 633,020, filed on April 11, 2024, the contents of which are incorporated herein by reference in their entirety. Background Art

[0003] Artificial intelligence (AI) models (e.g., machine learning (ML) models) can be used to generate output based on received natural language input prompts. Some AI models can be used to generate and output content (e.g., images) based on natural language input prompts. For example, a machine learning model can receive a user prompt that asks the model to "generate an image of a cat napping on a blanket." In response, the machine learning model can generate an image depicting a cat napping on a blanket. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Various embodiments according to the present disclosure will be described with reference to the accompanying drawings, in which:

[0005] Figure 1 An example of using a content generation system according to an embodiment of the present disclosure is shown;

[0006] Figure 2 An example of a content generation system according to an embodiment of the present disclosure is shown;

[0007] Figure 3 shows an example of a transformer block in a diffusion transformer model according to an embodiment of the present disclosure;

[0008] Figure 4A An example content generation system according to an embodiment of the present disclosure is shown;

[0009] Figure 4B shows example blocks included in a back-diffusion converter according to an embodiment of the present disclosure;

[0010] Figure 5 An example of a system for training a diffusion transformer according to an embodiment of the present disclosure is shown;

[0011] Figure 6 An example of a system for training an encoder model and / or a decoder model according to an embodiment of the present disclosure is shown;

[0012] Figure 7 An example of a process of using a content generation system according to an embodiment of the present disclosure is shown;

[0013] Figure 8An example of a process for training an encoder model and / or a decoder model according to an embodiment of the present disclosure is shown;

[0014] Figure 9 shows an example of a process for training a diffusion transformer model according to an embodiment of the present disclosure; and

[0015] Figure 10 is a simplified block diagram illustrating an example architecture of a system for training and / or using the models and systems described herein, according to certain embodiments. DETAILED DESCRIPTION

[0016] Various embodiments are described below. For illustrative purposes, specific configurations and details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments can be practiced without these specific details. Furthermore, well-known features may be omitted or simplified in order not to obscure the described embodiments.

[0017] There are challenges associated with machine learning (ML) models generating content in response to prompts. Improvements can be made in the content (e.g., images) that ML models can generate, how ML models are trained, and how inputs affect the content output generated by ML models. For example, improvements can enable efficient, high-quality, and accurate content generation in a scalable manner.

[0018] The use of transformer models in content generation models (e.g., image generation) is limited. This trend is reflected in the general preference for fully convolutional neural network architectures (e.g., U-Net) in diffusion models. The inductive bias of fully convolutional neural networks does not necessarily make them the optimal choice for diffusion models. Transformer models can be used as an alternative to fully convolutional neural network architectures in content generation (e.g., text-to-image generation).

[0019] Embodiments of the present disclosure relate to techniques for generating content given a prompt. The prompt may include text, video, audio, and / or images. The generated content may include images, videos, images with a particular style, images with a particular resolution, and / or images with a particular aspect ratio, and / or content with one or more other characteristics. Potential diffusion models can be used to generate content with high-resolution synthesis. By enabling large-scale, efficient (e.g., efficient use of resources (e.g., processing resources, memory resources, network resources)) and high-quality generation of content, embodiments can provide improvements over conventional systems, making embodiments useful for generative modeling.

[0020] Embodiments herein may construct a sequence of encodings comprising two modalities (e.g., text input and image input). The sequence may include positional encodings and flattened patches of latent pixel representations. After the patch encodings and text encodings are encoded and concatenated onto a common dimension, the disclosed inverse diffusion transformer may apply a series of modulated attentions and multi-layer perceptrons (MLPs) to generate content based on the text input (e.g., a prompt). Due to the conceptual distinction between the two different modalities (e.g., text and image encodings), embodiments may employ separate sets of weights for each modality. When using separate transformers for each modality, embodiments may combine sequences of each modality for attention operations (e.g., joint self-attention) such that the two representations can operate within their respective spaces while also taking each other into account.

[0021] The disclosed latent diffusion model uses separate weights for each modality (e.g., text modality and image modality) and / or is configured for bidirectional information flow, which can be referred to as a multimodal latent diffusion model (MMDiT). Compared to traditional techniques, MMDiT can improve text comprehension and spelling capabilities. For example, embodiments can more accurately render text in generated images, ensuring that text elements such as font, style, and size are appropriately presented. In addition, the MMDiT structure also enables the efficient and effective generation of high-quality images conditioned by text input.

[0022] Embodiments may include a rectified flow (RF) formulation that connects data and noise in a linear trajectory during training. Including RF can result in a straighter inference path, enabling sampling with fewer steps and therefore taking less time and using fewer resources (e.g., processing resources, network resources). Rectified flow can be used to train diffusion models.

[0023] Various embodiments can optimize memory usage by removing the use of one or more memory-intensive encoder models used during training at inference time. Removing one or more encoder models can significantly reduce memory requirements and can do so with minimal performance loss.

[0024] Embodiments can effectively handle multi-theme prompts, which include detailed descriptions of scenes, compositions, and / or situations involving more than one object, person, or concept. Multi-theme prompts provide embodiments with rich and complex information to generate corresponding content (e.g., images) that accurately represent the scenes or situations. Effectively handling multi-theme prompts requires embodiments (e.g., MMDiT) to understand and interpret the relationships between the different topics mentioned in the prompts in order to generate a coherent and realistic image. By effectively handling prompts, the creation of desired content that deviates minimally from the intended concept or situation can be improved, thereby reducing the resources consumed in the generation compared to previous techniques. For example, if the content generation is correct after the first prompt, rather than the second prompt, fewer network, memory, processing, and / or energy resources can be utilized.

[0025] Figure 1 An example of using a content generation system 108 according to various embodiments of the present disclosure is shown. The content generation system 108 can be used as part of a content creation system 100. The content creation system 100 may include a computing system 104, a network 106, and a content generation system 108. The content generation system 108 may receive prompts (e.g., natural language prompts) from the computing system 104 to cause content to be generated by one or more machine learning (ML) models 110. The generated content may be transmitted to the computing system 104 and presented by a user interface.

[0026] Computing system 104 may be a user device (e.g., a laptop, a personal computer, a phone, etc.). Computing system 104 may be a server. Computing system 104 may be capable of receiving input from user 102 via, for example, a user interface. In some embodiments, the input received by computing system 104 includes a prompt. This input may cause computing system 104 to transmit the prompt to content generation system 108 (e.g., via network 106). As an example, the user interface of computing system 104 may receive a natural language prompt (e.g., from user 102) describing desired features to be included in generated content, and the natural language prompt may be transmitted to content generation system 108 via network 106.

[0027] A prompt may include text (e.g., natural language text) describing desired characteristics of the content to be generated, such as one or more images and / or one or more videos. Characteristics may describe style, color, theme, mood, texture, contrast, depth, motion, saturation, focus, perspective, narrative, and / or other characteristics desired to be included in the generated content. A prompt may include at least one of text, audio, images, and / or videos. In some embodiments, the text may describe a scene (e.g., a scene from a book or screenplay) that can later be used to generate content corresponding to the text. In some embodiments, audio, image(s), and / or video(s) may be included in the prompt, causing the content generation system 108 to generate content corresponding to the audio, image(s), and / or video(s). For example, a prompt may include a portion of an image, and content may be generated that includes the portion or similar characteristics. In another example, a video scene from a movie may be included in the prompt, and the content generation system 108 may generate content that includes characteristics similar to the portion (e.g., similar style, color, theme, mood, texture, contrast, depth, motion, saturation, focus, perspective, narrative, etc.).

[0028] Hints or other information from computing system 104 may include information for determining one or more encoders to use. In an example, the encoders used to encode a hint may be predetermined and constant at runtime. In an example, a hint may explicitly specify which encoders or sets of encoders to use. In another example, the information included in the hint may be used by content generation system 108 to determine one or more encoders and / or one or more sets of encoders to use to encode a hint or a portion of a hint.

[0029] The prompts can be used as input to the content generation system 108 so that content can be generated. The content generation system 108 can use a set of one or more machine learning models 110 to generate content using the prompts. The set of one or more machine learning models 110 can include one or more encoder models, decoder models, and / or latent diffusion models (e.g., diffusion transformer models). The training and use of these models will be described in further detail below.

[0030] The generated content may include characteristics defined by the prompt. The content may include an image or video. The generated content may include one or more predefined characteristics. For example, the content may have predefined dimensions (e.g., pixel dimensions, number of pixels, bit size), and a predefined maximum size. The content generation system 108 may transmit the generated content to the computing system 104 for presentation (e.g., for display, for presentation as a downloadable file).

[0031] By presenting the content to user 102 using computing system 104, user 102 can view the content. Computing system 104 can store the content in memory, send the content to another computing system (e.g., a social media application, a different user device, etc.). In some embodiments, content generation system 108 may receive subsequent prompts (e.g., from computing system 104 or another computing system) that cause content generation system 108 to modify the generated content.

[0032] The network 106 can be configured to connect the computing system 104 and the content generation system 108, as shown. The network 106 can be configured to connect any combination of system components. In some embodiments, the network 106 is not part of the content creation system 100. For example, the content generation system 108 can run locally on the computing system 104, and / or the collection of one or more ML models 100 can run locally on the computing system 104.

[0033] Each of the network 106 data connections can be deployed via a public network (e.g., the Internet) or a private network (e.g., an intranet), where an access point, router, and / or another network node can communicatively couple the computing system 104 and the content generation system 108. The data connection between the components can be a wired data connection (e.g., a universal serial bus (USB) connector) or a wireless connection (e.g., a radio frequency-based connection). The data connection can also be made using a mesh network. The data connection can also provide a power connection. The power connection can provide power to the components to which it is connected. The data connection can provide data to and from the system components. One of ordinary skill in the art will recognize that devices can be communicatively coupled using a network (e.g., a local area network (LAN), a wide area network (WAN), etc.). In addition, devices can be communicatively coupled using a combination of wired and wireless methods (e.g., a wireless connection to a router, which is connected to a server via an Ethernet cable).

[0034] Interfaces between components communicatively coupled to content creation system 100, as well as interfaces between components within content creation system 100, can be implemented using network interfaces and / or application programming interfaces (APIs). For example, computing system 104 can implement a set of APIs for communicating with content generation system 108 and / or a user interface of computing system 104. In an example, computing system 104 uses a web browser in communicating with content generation system 108.

[0035] Figure 1The content creation system 100 shown in FIG can further implement the steps S120 to S126 shown. The steps shown can be implemented by executing instructions stored in the memory of the content creation system 100, wherein the execution of the instructions is performed by the processor of the content creation system 100.

[0036] At step S120, a prompt may be transmitted from computing system 104 to network 106. The prompt may include information received from a user interface of computing system 104. For example, user 102 may have entered: "Please create an image of an old, rusty robot wearing pants and a jacket skiing in a supermarket," and the prompt may reflect the entered information and be transmitted to network 106.

[0037] At step S122, the prompt may be transmitted from the computing system 104 to the content generation system 108 via the network 106. After the content generation system 108 receives the prompt, the content generation system 108 may use one or more machine learning models 110 to generate content using the prompt.

[0038] At step S124 , the content generation system 108 may transmit the generated content to the network 106 .

[0039] At step S126, network 106 may transmit the generated content to computing system 104. Once computing system 104 receives the generated content, computing system 104 may present the generated content or a portion thereof using a user interface of computing system 104. For example, computing system 104 may present an image or video on a display viewable by user 102.

[0040] Figure 2 1 shows an example of a content generation system 108 according to various embodiments of the present disclosure. The content generation system 108 may be about Figure 1 The content generation system 108 is described. The content generation system 108 can be configured to receive hints 208 and output generated content 230. The content generation system 108 can include one or more coding models, a coded addition system 214, an inverse diffusion transformer 224, and a decoder model 228. The one or more coding models can include one or more hint coding models and a time-step coding model 204.

[0041] Hints 208 may be transmitted from a computing system (e.g., computing system 104 described above). Hints 208 may be received from the system (e.g., via a network). Hints 208 may be received by a user interface of the system. Hints 208 may describe desired characteristics of the content to be generated by content generation system 108. For example, dimensions (e.g., pixel dimensions, number of pixels, bit size), style, color, theme, mood, texture, contrast, depth, motion, saturation, focus, perspective, narrative. Hints 208 may be received by one or more of one or more hint encoding models.

[0042] A prompt encoding model in the set of prompt encoding models can be configured to represent a prompt 208 or a portion of a prompt 208 in a multi-dimensional space (e.g., a vector space). The prompt encoding model can include a neural network layer for converting a prompt 208 or a portion of a prompt 208 into a prompt encoding (e.g., first prompt encoding(s) 212, prompt condition 220) in a high-dimensional space. The neural network layer used to generate the prompt encoding can be referred to as an embedding layer. The prompt encoding model can be configured and / or pre-trained to generate encodings of prompts represented as text, audio, images, and / or video. The prompt encoding model can be a joint image and text encoding model (e.g., a contrastive language-image pre-training (CLIP) model), a text encoder from a CLIP model, a large language model, a T5 model, a convolutional neural network transformer, or a recurrent neural network. One of ordinary skill in the art, having benefit of this disclosure, will recognize that other ML models can also be used for prompt encoding.

[0043] The hint encoding model may include a first hint encoding model set 210 and / or a second hint encoding model set 218. The hint encoding model set may include one or more hint encoding models. The hint encoding model may include one or more frozen hint encoding models (e.g., trainable model properties are preserved). The hint encoding model set used to encode the hint 208 or a portion of the hint 208 may be determined based on the hint 208. For example, the hint encoding model set used to encode the hint 208 or a portion of the hint 208 may be determined based on instructions in the hint 208 (e.g., to use a specific hint encoding model set). In an example, the hint encoding model set used to encode the hint 208 or a portion of the hint 208 may be determined by information included in the hint 208 (e.g., whether the hint includes text, the hint includes text and an image, the hint includes audio, etc.). The hint encoding model set used to encode the hint 208 or a portion of the hint 208 may be predefined (e.g., specified by a system administrator). The hint encoding model set used to encode the hint 208 or a portion of the hint 208 may be determined based on instructions received by the computing system.

[0044] In some embodiments, the hint 208, or a portion of the hint 208, is received by a first hint coding model set 210 and / or a second hint coding model set 218. The first hint coding model set 210 may include one or more hint coding models for generating an encoding of at least a portion of the hint 208. The encodings generated by the first hint coding model set 210 may be combined (e.g., via concatenation) into a single vector space represented by a combined encoding 212. The first hint coding model set 210 and the second hint coding model set 218 may include one or more identical hint coding models (e.g., a common CLIP model). The first hint coding model set 210 and the second hint coding model set 218 may include different numbers of hint coding models. The second hint coding model set 218 may include one or more hint coding models to generate an encoding of at least a portion of the hint 208. The encodings generated from the second hint coding model set 218 may be combined (e.g., via concatenation) into a single vector space represented by a hint condition 220. The hint condition 220 vector space may have the same dimensionality as the noise latent space 222 input to the inverse diffusion transformer 224.

[0045] Time step encoding model 204 can be used to encode time step 202 into encoded time step 206. Time step encoding model 204 can receive time step 202. Time step 202 can represent a time step in a backdiffusion process. Time step encoding model 204 can encode time step 202 using a neural network and / or based on a function. For example, time step encoding model 204 can use a sine function to determine encoded time step 206 based on time step 202. The output of the sine function can be represented in a vector space as encoded time step 206. The vector space of encoded time step 206 can have the same dimensions as combined code 212.

[0046] The code addition system 214 can add the encoded time step 206 vector to the combined code 212 to generate a temporal condition 216. The temporal condition 216 can be used by the modulated attention mechanism of the inverse diffusion transformer 224 and can enable conditional generation. When the time step 202 used to generate the temporal condition 216 is closer to the middle of the time window (e.g., a middle time step) than other time steps farther from the middle of the time window, the temporal condition 216 can be given a higher weight.

[0047] The inverse diffusion transformer 224 can receive the temporal condition 216, the prompt condition 220, and the noise latent space 222 as inputs. The inverse diffusion transformer 224 can use these inputs to generate a conditional latent space 226. The noise latent space 222 can be a latent space containing randomly generated noise. The noise latent space 222 can be generated based on sample values ​​according to a distribution (e.g., a Gaussian distribution). The noise latent space 222 can be generated based on a seed. The seed can be input to the content generation system 108 (e.g., via a user interface). The noise latent space 222 can be stored in a memory and used by the inverse diffusion transformer 224.

[0048] Noise latent space 222 may include position information. In some embodiments, noise latent space 222 is generated by adding a position embedding to an initial noise latent space. The initial noise latent space may have been generated using the techniques described above with respect to noise latent space 222. The initial noise latent space may represent a pixel encoding. The position embedding may add information about the position of elements in noise latent space 222. The position embedding may help the inverse diffusion transformer understand the relative positions and relationships between different parts of the image.

[0049] Inverse diffusion transformer 224 can be a machine learning model trained to generate a conditioned latent space (e.g., conditioned latent space 226) using a noise latent space (e.g., noise latent space 222). Techniques for training inverse diffusion transformer 224 are described in further detail below. Conditioned latent space 226 can be generated using a combination of cue conditions 220, temporal conditions 216, and noise latent space 222.

[0050] The inverse diffusion transformer 224 can generate a conditioned latent space 226 by removing noise from the noisy latent space 222. The inverse diffusion transformer 224 can obtain the conditioned latent space 226 by iteratively removing noise from the noisy latent space 222 at time steps (e.g., time step 202). The inverse diffusion transformer 224 can generate the conditioned latent space 226 using one or more transformer blocks, which will be described in further detail below. The conditioned latent space 226 can be considered an encoded form of content (e.g., generated content 230). The conditioned latent space 226 can be stored in a memory of the content generation system 108.

[0051] The decoder model 222 can receive the conditioned latent space 226 as input and generate content 230 using the conditioned latent space 226. The decoder model 228 can be trained using the techniques described further below. The decoder model 228 can be configured to receive the conditioned latent space 226 after it is output from the backdiffusion transformer 224. The decoder model 228 may include neural network layers for generating content from an encoding of the content (e.g., the conditioned latent space 226). The decoder model 228 may include a recurrent neural network, a long short-term memory network, a transformer model, a convolutional neural network, or other model architectures. Those of ordinary skill in the art, having benefit of this disclosure, will recognize other architectures that can be used as the decoder model 228.

[0052] Figure 3 An example of a converter block 300 of a diffusion converter model (e.g., the reverse diffusion converter 224) according to various embodiments of the present disclosure is shown. Converter block 300 can be one of a plurality (e.g., 15, 38, or many) of converter blocks included in the diffusion converter model. Converter block 300 can receive inputs from other converter blocks and / or transmit outputs to other converter blocks.

[0053] Transformer block 300 may receive as inputs the hint conditions 220, the noise latent space 222, and the temporal conditions 216 (e.g., the hint conditions 220, noise latent space 222, and temporal conditions 216 described above). If transformer block 300 is not the first transformer block in the inverse diffusion transformer, transformer block 300 may receive inputs from a previous transformer block. Transformer block 300 may receive inputs from a coding-additive system (e.g., coding-additive system 214), a second hint coding model set (e.g., second hint coding model set 218), and a noise latent space generation system. Transformer block 300 may use these inputs to generate conditional hints 320 and / or conditional latent space 322. The conditional latent space may be conditional latent space 226 described above. Transformer block 300 may transmit the conditional hints 320 and conditional latent space 332 to subsequent transformer blocks. The transformer block 300 may transfer the conditioned latent space 322 to a decoder model (eg, decoder model 228 ).

[0054] As part of the operations performed, the transformer block 300 may perform operations on the prompt condition 220 vector and the noise latent space 222 vector, respectively. Because the prompt condition 220 vector and the noise latent space 222 vector can represent encodings that include many conceptual differences (e.g., image encoding and text encoding), a series of operations may be performed on each of these vectors. For example, the prompt condition 220 may be subjected to a layer normalization (layer normalization) 302a operation, a modulation 304a, and / or a linearization 306a operation to generate a first intermediate prompt value 324.

[0055] Layer normalization 302a can be used to make neurons in a common layer have the same normalization terms (e.g., the same mean and the same variance). Layer normalization 302a can enable smoother gradients, faster training, and higher accuracy by normalizing the distribution of intermediate layers. Adaptive layer normalization (adaLN) can be used to condition diffusion networks on text representations to enable efficient parameter adaptation.

[0056] The modulation mechanism can enable conditional generation. Modulation can use time conditions 216. The modulation mechanism can use scaling (e.g., adjusting the range of the data) and shifting (e.g., shifting the data distribution) operations. Scaling and shifting operations can make the characteristics of the data more suitable for modeling.

[0057] The linear 306a operation can perform a linear transformation on the input. The linear 306a operation can be used to clean data, extract features from the data, and / or prepare the data for further operations. The linear 306a layer operation can come from training using a learnable low-rank (LoRA) matrix.

[0058] Similar to layer normalization 302a, modulation 304a, linearization 306a, layer normalization 302, modulation 304b, and / or linearization 306b operations can be performed on the noise latent space to generate a first intermediate noise value 326. After respectively operating on the cue condition 220 vector and the noise latent space 222 vector, the first intermediate cue value 324 and the first intermediate noise value 326 generated by the corresponding operations can be combined (e.g., concatenated). The combination can be performed by a concatenation system 308, which is configured to concatenate the two vectors.

[0059] The combined vector can be used by the joint self-attention system 310 to generate an output. The joint self-attention system 310 combines the sequence of first intermediate cue values ​​324 and first intermediate noise values ​​326 (e.g., from different modalities) for the attention operation, so that the two representations operate within their respective vector spaces while also taking into account each other. The joint self-attention system 310 enables the contextual relationship between the two intermediate embedding spaces to be captured. The output of the joint self-attention system 310 can be operated on using two separate sequences of operations. For example, each sequence of operations can include any combination of linear transformations, layer normalization, modulation, code addition, and / or multi-layer perception (MLP). The MLP can be configured to perform multiple layers of nonlinear transformations on the input. An exemplary first sequence of operations performed on the output from the joint self-attention system 310 is shown as: linear operation 312a, code addition using code addition system 214, code addition using code addition system 214, layer normalization 314a, modulation 316a, MLP 318a, and code addition using code addition system 214. An exemplary second sequence of operations performed on the output from the joint self-attention system 310 is shown as: linear operation 312a, coded addition using the coded addition system 214, layer normalization 314a, modulation 316a, MLP 318a, and coded addition using the coded addition system 214.

[0060] After operating on the output of the joint self-attention system 310 with two separate sequences of operations, the conditioned cues 320 and conditioned latent space 322 generated by the respective sequences of operations can be output from block 300. The separate sequences of operations can be equivalent to a transformer with independent weights for each modality, but allow each of the two transformers to perform operations informed by the other.

[0061] As an example, the conditional prompt 320 can be generated from the following: a joint self-attention system, a first sequence of operations (e.g., 302a, 304a, 306a, 308, 312a, 214, 314a, 316a, and / or 318a), and at least one first weight included in a first weight set associated with a first domain (e.g., a text domain) of the conditional prompt 320. The conditional latent space 322 can be generated from the following: a joint self-attention system, a second sequence of operations (e.g., 302b, 304b, 306b, 308, 312b, 214, 314b, 316b, and / or 318b), and at least one second weight included in a second weight set associated with a second domain (e.g., a content domain) of the conditional latent space 322.

[0062] Figure 4A An example content generation system 400 is shown in accordance with various embodiments of the present disclosure. Figure 4B1 shows an example converter block 414 (e.g., 414a, 414b, 414d) included in the inverse diffusion converter 224 according to various embodiments of the present disclosure. Example converter block 414 is an example of converter block 300. Similar components may be represented by similar part numbers. For example, the linear 404b operation may be similar to the linear 404a operation.

[0063] The example content generation system 400 can receive the hints 208 (as described above) and generate content 230 (as described above) based on the hints 208. The example content generation system 400 includes a first hint encoding model set 210, a second hint encoding model set 218, a time step encoding model 204, a back-diffusion transformer 224, and a decoder model 228, each of which is at least about Figure 2 Described in the previous article.

[0064] The first set of hinted encoding models 210 can generate one or more encodings for the hint 208. In the example shown, the first set of hinted encoding models 210 includes the CLIP-G / 14 encoding model and the CLIP-L / 14 encoding model. In some embodiments, more or fewer encoding models are used. In some embodiments, different combinations of encoding models are used. The first set of hinted encoding models 210 can include one or more encoding models used during training of the example content generation system 400. In the example shown, each encoding model included in the first set of hinted encoding models 210 generates an encoding for at least a portion of the hint 208.

[0065] The example content generation system 400 illustrates an embodiment where the encoding outputs of the CLIP-G / 14 encoding model and the CLIP-L / 14 encoding model can be pooled and have sizes of 768 and 1280, respectively. The pooled outputs can be concatenated to obtain the vector condition c vec This is shown as combined encoding 212. Combined encoding 212 may include coarse-grained information about hint 208.

[0066] The penultimate hidden representations from the CLIP-G / 14 encoding model and the CLIP-L / 14 encoding model can be concatenated per channel as CLIP context conditions Then, c can be encoded as the final hidden representation of the T5-v1.1-XXL model Then, the Zero-padded to 4096 dimensions to match the T5 representation and parallelized along the sequence axis. Concatenate to obtain the final context representation c ctxt Shown as prompt condition 220 .

[0067] A second set of hinted encoding models 218 can generate one or more encodings for prompt 208. In the example shown, the second set of hinted encoding models 218 includes the CLIP-G / 14 encoding model and the CLIP-L / 14 encoding model, as well as the T5 XXL encoding model. In some embodiments, more or fewer encoding models are used. In some embodiments, different combinations of encoding models are used. The second set of hinted encoding models 218 can include one or more encoding models used during training of the example content generation system 400. In the example shown, each encoding model included in the second set of hinted encoding models 218 generates an encoding for at least a portion of prompt 208.

[0068] The example content generation system 400 illustrates that, in addition to the operations described above, the prompt conditions 220 may also be subjected to a linear operation before being used by the inverse diffusion transformer 224. The linear operation has been described above with respect to the linear 306a operation. The example content generation system 400 illustrates that the combined code 212 may be input to the MLP 402a. The operation of the MLP has been described in further detail above. The output of the MLP 402a may be transmitted to the code addition system 214 to be added to the code time step 206 output from the time step code model 204 to generate the temporal condition 216 (described in further detail above).

[0069] The encoded time step 206 can be generated by the time step encoding model 204. The time step encoding model 204 can use the received time step 202 to generate the encoded time step 206. The time step encoding model 204 can perform sinusoidal encoding 406 and use an MLP 402b. The MLP 402b can operate in a manner similar to the MLP described above. The sinusoidal encoding 406 can include using a sinusoidal function to determine an output code based on the time step 202. The output of the sinusoidal encoding 406 can be represented in a vector space.

[0070] The noise latent space 222 may also be received and / or generated by the example content generation system 400 for use as an input to the inverse diffusion transformer 224. The noise latent space 222 may be a latent pixel representation. The noise latent space 222 may be generated by adding (e.g., using the code addition system 214) a positional encoding 412 to the initial noise latent space 408 or a latent space derived using the initial noise latent space 408 (e.g., after the initial noise latent space 408 has been subjected to a block 410 operation and / or a linear 404b operation). The block 410 operation may be used to convert the noise latent pixel representation into a latent pixel representation. The blocks (eg, 2×2 blocks) are flattened into block encoding sequences of length 0.5*h*0.5*w. The noise latent space 222 may have the same dimensions as the prompt condition 220.

[0071] The inverse diffusion transformer 224 may use the temporal condition 216, the cue condition 220, and the noise latent space 222 as input to generate content 230. The inverse diffusion transformer 224 may include any number of transformer blocks 414 (e.g., MM-DiT-blocks 414a, 414b to 414d). In an embodiment, the size of the inverse diffusion transformer 224 may be parameterized with respect to the depth d of the model (e.g., the number of attention blocks) by setting the hidden size to 64*d (expanded to 4*64*d channels in the MLP block), and the number of attention heads is equal to d. The inverse diffusion transformer 224 and the transformer block 414 may use the temporal condition 216 and the cue condition 220 as input to the modulation mechanism(s). The transformer block 414 will be further described below (e.g., with respect to Figure 3 and Figure 4B ).

[0072] After a number of blocks included in the inverse diffusion transformer 224 have received inputs and generated corresponding outputs, the final transformer block 414d can output the conditioned latent space 322. The conditioned latent space 322 can be output from the inverse diffusion transformer 224. The conditioned latent space 322 can be subjected to a modulation 416 operation (e.g., the modulation operation described above), a linear 404c operation, and / or an unpatching 418 operation. The unpatching 418 operation upsamples the conditioned latent space 332 to a dimension that the decoder model 228 (e.g., described above) can use to generate the content 230.

[0073] Figure 4B 4 shows an example block 414 (e.g., 414a, 414b through 414d) included in the inverse diffusion transformer 224 according to various embodiments of the present disclosure. The example transformer block 414 can receive the temporal condition 216, the cue condition 220, and / or the noise latent space 222 as inputs and use these inputs to generate the conditioned cue 320 and / or the conditioned latent space 322. The conditioned cue 320 and the conditioned latent space 322 are described in detail above and can be used in subsequent transformer blocks (e.g., the example transformer block 414d) and / or in the output of the transformer model (e.g., the inverse diffusion transformer 224). The conditioned latent space 322 can be decoded by the decoder model to generate the content 230.

[0074] The example transformer block 414 can perform a swish activation function (SiLu) 420a operation and a linear 404d operation on the temporal condition 216 to generate a first temporal condition result 424a. The first temporal condition result 424a can be used to condition the vector space being operated to generate the conditioned prompt 320. Similarly, operations (SiLu420b and linear 404e) can be performed on the temporal condition 216 to generate a second temporal condition result 424b. Since different weight configurations (e.g., based on training) may be used compared to the prompt modality (e.g., text, prompt condition 220) and the content modality (e.g., image, noise potential space 220), the first temporal condition result 424a and the second temporal condition result 424b may be different. As shown, the values ​​of the first temporal condition result 424a and the second temporal condition result 424b can be used in each step of the operation performed by the example transformer block 414.

[0075] Before being used as input to the stitching system 308, the prompt conditions 220 can undergo other operations such as layer normalization 302a operation, modulation 304a operation, linearization 306a operation, and / or root mean squared (RMS) normalization operation 422. The stitching system can use the input values ​​to generate values ​​to be transmitted to the joint self-attention system 310. The joint self-attention system 310 can generate outputs using inputs associated with the prompt conditions 220 and the noise latent space 222. The normalization operation 422 can prevent the attention logit function from growing instabilities and enable efficient training. When the model uses the normalization operation 422 in addition to the layer normalization 302a operation, the normalization operation 422 can help stabilize large models.

[0076] The output of the joint self-attention system 310 can be used to perform a linear 312a operation on the conditioned hint 320, a coded multiplication operation using the coded multiplication system 426, a coded addition operation using the coded addition system 214, a layer normalization 314a operation, a modulation 316a operation, an MLP 318a operation, an additional coded multiplication operation using the coded multiplication system 426, and / or a coded addition operation using the coded addition system 214. The coded multiplication system can generate an output using values ​​generated using the temporal conditions 216 and intermediate values ​​from the sequence of operations. The coded addition system 214 can generate an output using the hint conditions 220 and intermediate values ​​from the sequence of operations.

[0077] Similar operations can be performed on the noise latent space 222 as on the prompt condition 220, but the prompt condition 220 needs to be replaced by the noise latent space 222. Similar operations can generate the conditioned latent space 322.

[0078] Figure 5 An example of a system 500 for training a diffusion transformer according to various embodiments of the present disclosure is shown. The diffusion transformer system may include a forward diffusion transformer 520 (e.g., a system or model) and a reverse diffusion transformer 224 that may be configured by a transformer adjustment system 518 during a training process. The forward diffusion transformer 520 may include a system configured to perform a process of adding noise to potential training content 518 (e.g., potential ground truth) to generate noisy potential training content 522. On the other hand, the reverse diffusion transformer 224 may be configured to generate potential output content 526 using the encoded prompt conditions 506, the temporal conditions 536, and / or the noisy potential training content 522. The reverse diffusion transformer may be the reverse diffusion transformer 224 described above.

[0079] In some embodiments, the architecture of the forward diffusion transformer 520 may be similar to that described above with respect to Figure 2 The architecture is similar to the inverse diffusion transformer model described in , but is configured to use latent training content 518 to generate noisy latent training content 522 .

[0080] Before the backdiffusion transformer 224 can be used during inference time, the backdiffusion transformer 224 can first be trained to restore the noise introduced by the forward diffusion transformer 520. The forward diffusion transformer 520 can introduce noise to the latent training content 518 to generate noisy latent training content 522, so that the backdiffusion transformer 224 can learn how to restore the noise introduced by the forward diffusion transformer 520. The forward diffusion transformer 520 uses the latent training content 518 generated by encoding the training content 514 using the encoder model 516. The latent training content 518 can be considered as the ground truth for training purposes. The forward diffusion transformer 520 generates noisy latent training content 522 with progressively more noise and passes the generated noisy latent training content 522 to the backdiffusion transformer 224 to undo the added noise and attempt to obtain the latent training content 518 from the noisy latent training content 522. The forward diffusion transformer 520 can add noise to the latent training content 512 by sampling a Gaussian distribution to obtain a vector of the same size as the latent training content 518, and then interpolating between the latent training content 518 and the noise data based on coefficients derived from the sampled time step values.

[0081] Backward diffusion transformer 224 and / or forward diffusion transformer 520 may be trained using a training dataset comprising content pairs (e.g., training content 514) and prompts (e.g., training prompts 502). For example, the training data in the training dataset may include training prompt 502, which states, "Picture of a bear wearing a suit and a top hat, holding a sign that reads 'I can't stand it', in a river in the middle of a forest." and corresponding training content 514, which includes an image of a bear wearing a suit and a top hat, holding a sign that reads 'I can't stand it', in a river in the middle of a forest. The training data may have been generated by performing a clustering-based deduplication method to remove perceptual and semantic repetitions in the initial training data.

[0082] The training content 514 and the training hints 502 can each be used during the process of training the inverse diffusion transformer 224. The training hints 502 included in the training data can also be used during training of the hinted coding models included in the first hinted coding model set 508 and / or the second hinted coding model set 504 (e.g., before the first hinted coding model set 508 and / or the second hinted coding model set 504 are used to perform inference-time computations to generate the hinted conditions 506, combined encodings 510, combined encodings 212, and / or hinted conditions 220 for the encodings).

[0083] The training content 514 can be used with the encoder model 516 to generate latent training content 518. The forward diffusion transformer 520 can use the latent training content 518 to generate noisy latent training content 522. The noisy latent training content 522 can be received from the encoder model 516. The encoder model 516 can have been trained to generate an output latent representation of the input content received by the encoder model 516. The latent training content 518 can be a high-dimensional representation of the training content 514. The encoder model 516 can have been pre-trained for general content or trained for a specific content form (e.g., images, videos, images of a specific style, images of a specific emotion, images of a specific color, etc.).

[0084] At least a portion of the training cues 502 included in the training data can be input to a first set of cued coding models 508 and / or a second set of cued coding models 504. The first set of cued coding models 508 can be the first set of cued coding models 210 described above or another set of cued coding models. The second set of cued coding models 504 can be the second set of cued coding models 218 described above or another set of cued coding models. One or more cued coding models included in the set of cued coding models can have been trained to generate encodings of cues (e.g., training cues 502) or portions of cues. The training cues 502 can include text, audio, video, and / or images, among other things. The encoding cued conditions 506 generated by the second set of cued coding models 504 can be used by the inverse diffusion transformer 224 to generate potential output content 526. The combined encoding 510 generated by the first set of cued coding models 508 can be used by the code addition system 214 to generate temporal conditions 536, which can be used by the inverse diffusion transformer 224 to generate potential output content 526. The code addition system 214 may be the code addition system 214 described above and adds the combined code 510 and the encoded time step 534 .

[0085] The encoded time steps may be determined by the time step encoding model 204 (e.g., Figure 2 and Figure 4A The time step encoding model 204 described above is generated based on the time steps 512. The time steps can be sampled in a non-uniform manner. In some embodiments, the time steps are sampled at a higher frequency between the starting time step and the ending time step. For example, the sampled time steps can follow an approximately normal distribution or a logit-normal distribution. The time steps and the time step encoding model are described in further detail herein. By sampling at a higher frequency at the intermediate steps, the generated output can be more accurate (e.g., more accurately reflect the prompt) compared to earlier and later steps where the backdiffusion process is very difficult or very easy.

[0086] The noise potential training content 522 generated by the forward diffusion transformer 520 can also be used as an input to the reverse diffusion transformer 244 to train the reverse diffusion transformer 224 to generate potential output content 526 based on the noise potential training content 522. The reverse diffusion transformer 224 can use the encoded prompt conditions 506, the time conditions 536 and the noise potential training content 522 to learn how the recognition training prompts 502 correspond to the potential training content 518 used to generate the noise potential training content 522. The reverse diffusion transformer 224 can use the encoded prompt conditions 506 and the time conditions 536 to perform conditioning (e.g., cross-attention conditioning, self-attention conditioning). Learning / training can also be performed in multiple iterations. In the iteration, the parameter values ​​of the reverse diffusion transformer 224 and / or the forward diffusion transformer 520 (e.g., if the forward diffusion transformer model system is being used) can be adjusted using the transformer weight adjustment signal 530 from the transformer adjustment system 528.

[0087] The transformer adjustment system 528 can compare the potential output content 526 generated by the back-diffusion transformer 224 with the potential training content 518 (e.g., ground truth) to determine a transformer weight adjustment signal 530 to transmit to the forward diffusion transformer 520 (e.g., in an embodiment where the forward diffusion transformer 520 is a model) and / or the back-diffusion transformer 224. The transformer adjustment system 528 can use a loss function to compare the potential training content 518 and the potential output content 526. The transformer adjustment system 528 can transmit the transformer weight adjustment signal 530 with the goal of minimizing the loss function. The loss is used to generate gradients used to train the transformer(s) during back-propagation.

[0088] In some embodiments, the inverse diffusion transformer 224 uses an image of a first size and performs fine-tuning using an image of a second size that is larger than the first size and / or an image with a mixed aspect ratio.

[0089] In some embodiments, the training prompts 502 and training content 514 are obtained from a predefined training dataset. In some embodiments, the predefined training dataset may be limited in size and / or in the properties of the training prompts 502 corresponding to the training content 514. At least a portion of the training data and training content 514 may be used to synthetically generate the training prompts 502. A visual language model (e.g., CogVLM) may be used to generate the synthetic training prompts 502. Training the inverse diffusion transformer 224 may include training using the generated / synthesized training prompts 502 and at least a portion of the corresponding training content 514. In some embodiments, the training data includes prompts from the original dataset and synthetically generated using the techniques described above. The training data may include a split (e.g., a 50 / 50 split) between the original dataset and the synthetic prompts. The synthetically generated prompts may be used to improve the image output generated by the inverse diffusion transformer 224. Synthetic captions may enable models that generate images using text to be trained on more data and / or on data with varying characteristics. For example, manually generated prompts may include different language and / or focus on different descriptions than prompts generated by a visual language model.

[0090] Figure 6 An example of a system 600 for training an encoder model (e.g., encoder model 516) and / or a decoder model (e.g., decoder model 228) according to various embodiments of the present disclosure is shown. The system 600 may include the encoder model 516, the decoder model 228, and an autoencoder adjustment system 608. The decoder model 228 and / or the encoder model 516 may be trained by using the autoencoder adjustment system 608 to compare ground truth training content 602 with output content 606 generated by the decoder model 228, and adjusting the weights of the encoder model 516 and / or the decoder model 228 based on the comparison. Through training iterations, the decoder model 228 may learn to generate accurate output content 606 using the latent training content 604. The latent training content 604 may be compared to the latent training content 604 generated by the decoder model 228 described above. Figures 1 to 5 The conditioned latent spaces (e.g., conditioned latent space 226, potential output content 526) generated by the described inverse diffusion transformer model have the same dimensions so that the decoder model 228 can use the conditioned latent space 226 generated by the inverse diffusion transformer 224 to generate content 230.

[0091] The encoder model 516 may have been trained by the system 600 or be trained by the system 600 to generate embeddings of the ground truth training content 602. The training content 602 may be an image, video, or other content. The training content 602 may include the training content used for the above training (e.g., in pixel space). The encoder model 516 processes the training content 602 through a series of convolution blocks that each perform downsampling. All convolutions may be parameterized in the form of weight normalization. The encoder model 516 may map the input to a lower dimensional space. Increasing the number of potential channels may improve reconstruction performance. The reconstruction quality of the encoder model 516 may provide an upper limit for the image quality achievable after potential diffusion training. As an example, the number of potential channels may be equal to 16 and achieve an improvement over an encoder model using 8 channels. The number of channels may be balanced when considering that a lower number of channels allows the model to use fewer resources. There may be less reward when the number of potential channels is increased.

[0092] The latent training content 604 generated by the encoder model 516 can be transmitted to the decoder model 228 to be used as input to the decoder model 228. The decoder model 228 can be trained by the system 600 to output generated output content 606 (e.g., in pixel space) based on the latent training content 604 (e.g., encoding of the ground truth training content 602). The output content 606 can include pictures, videos, or other content. The architecture of the decoder model 228 can be similar to that of the encoder model 516, but with upsampling blocks. The architecture of the decoder model 214 is described above with respect to Figure 3 is further described.

[0093] The autoencoder adjustment system 608 can use the output content 606 and the training content 602 (reference facts) to confirm the weight adjustment signal 610 to be sent to the encoder model 516 and / or the decoder model 228. The autoencoder adjustment system 608 can compare the output content 606 and the training content 602 using a loss function. In some embodiments, a reconstruction loss function is used. Based on the comparison of the output content 606 and the training content 602 using the loss function, the autoencoder adjustment system 608 can transmit the weight adjustment signal 610 to the encoder model 516 and / or the decoder model 228, with the goal of reducing the loss function. In some embodiments, using a convolutional discriminator model, the autoencoder adjustment system 608 uses an adversarial loss term. The discriminator model may include hyperparameters.

[0094] Figure 7 An example of a process 700 for using a content generation system (such as the content generation system 108 described above) is shown according to various embodiments of the present disclosure.

[0095] At step 702, the content generation system receives a prompt. The prompt can be received from a computing system (e.g., computing system 104). The prompt can describe one or more desired characteristics of the content to be generated by the content generation system. For example, the desired characteristics may include style, color, theme, mood, texture, contrast, depth, motion, saturation, focus, perspective, narrative, and / or other characteristics. The prompt can include example content (e.g., images, videos), audio, text, images, videos. The content in the prompt can be used as inspiration for generating content. The prompt can include content to be added and / or changed by the content generation system.

[0096] At step 704, a first hint encoding (e.g., combined encoding 212) may be generated. The first hint encoding may be generated by inputting the hint into a first set of one or more coding models. The coding models included in the first set of one or more coding models may each generate an encoding for at least a portion of the hint and output the encoding for that portion, which may then be combined with other encodings of other coding models from the first coding model set to generate the first hint encoding. The first coding model set may include one or more text encoders. The first coding model set may be used for one or more portions of the hint. As an example, the first coding model set may include a CLIP-G / 14 model and a CLIP-L / 14 model.

[0097] At step 706, a second prompt encoding (e.g., prompt condition 220) can be generated. The second prompt encoding can be generated by inputting the prompt into a second set of one or more coding models. The coding model included in the second set of one or more coding models can each generate a coding of at least a portion of the prompt, and output the coding of the portion, which can then be combined with other codings of other coding models from the second coding model set to generate the second prompt encoding. The second prompt coding model set can include one or more text encoders. The second coding model set can be used for one or more parts of the prompt. As an example, the second coding model set can include CLIP-G / 14 model, CLIP-L / 14 model, and / or T5 XXL model.

[0098] The first set of coding models and the second set of coding models may include one or more of the same coding models (e.g., having the same parameters and architecture). The first set of coding models and the second set of coding models may include one or more coding models that are not included in the second set of coding models and the first set of coding models, respectively. In some embodiments, the first set of coding models and / or the second set of coding models are trained using more coding models than the corresponding set used at inference time.

[0099] At step 708, a first conditional latent space and a second conditional latent space are generated. The first conditional latent space can be referred to as a conditional cue (e.g., conditional cue 320). The first conditional latent space can be generated by a first transformer block in a diffusion transformer model. The first transformer block can generate the first conditional latent space based on the first cue encoding, the second cue encoding, and a noise latent space. The noise latent space can share a common dimension with the second cue encoding.

[0100] The first hint encoding or an encoding derived therefrom can be combined (e.g., added) with the encoding of the time step to generate a temporal condition signal (e.g., temporal condition 216). The temporal condition signal can include the encoding of the time step. The first conditioned latent space can be generated by the first transformer block using the temporal condition signal (e.g., temporal condition 216).

[0101] The second conditional latent space may be referred to as a conditional latent space (e.g., conditional latent space 322). The second conditional latent space may be generated by the first transformer block in the diffusion transformer model. The first transformer block may generate the second conditional latent space based on the first hint encoding, the second hint encoding, and the noise latent space. The second conditional latent space may be generated by the first transformer block using a temporal conditional signal.

[0102] The first transformer block can perform a back-diffusion transform on the first hint encoding and the noise latent space. The back-diffusion operation can be informed by a temporal conditioning signal. The first transformer block can jointly operate on the intermediate values ​​generated from the first hint encoding and the noise latent space using a joint self-attention system (e.g., joint self-attention system 310). The output of the joint self-attention system can be further operated on in two independent series of operations to generate a first conditioned latent space and a second conditioned latent space in the corresponding domains. The independent series of operations performed can use independent weights (e.g., different weights).

[0103] The output from the first transformer block can be used as the input to the subsequent transformer block. Any number of transformer blocks can be chained together within the diffusion transformer model. The transformer module can include a linear layer trained using a learnable low-rank (LoRA) matrix. The last block in the chain is described with reference to step 710.

[0104] At step 710, a third conditional latent space is generated. The third conditional latent space can be generated by a second transformer block included in the diffusion transformer model. The second transformer block can have an architecture similar to the first transformer block. The third conditional latent space can be generated based on the output of the previous transformer block. For example, the third conditional latent space can be generated based on the first conditional latent space and the second conditional latent space. The third conditional latent space can also be generated based on a temporal conditioning signal obtained by encoding the current time step. Since the second transformer block can have an architecture similar to the first transformer block, the second transformer block can be configured to generate a conditional cue signal and a conditional latent space signal. The second transformer block can output a conditional latent space signal to be transmitted to a decoder model (e.g., decoder model 228). The conditional latent space signal can include an encoded representation of the content described in the cue.

[0105] At step 712, a decoder model may be used to decode the encoded conditioned latent space representation of the content described in the prompt. The content may include one or more characteristics described in the prompt.

[0106] In some embodiments, after generating content, the content generation system may receive a subsequent prompt, causing the generation of second content that is different from the first content and based on the first prompt and / or the first content. For example, the second prompt may request the addition, further enhancement, removal, or change of one or more features.

[0107] Figure 8 An example of a process 800 for training an encoder model (e.g., encoder model 516, time-step encoding model 204) and / or a decoder model (e.g., decoder model 228) according to various embodiments of the present disclosure is shown. The encoder and / or decoder models may be trained using the process 800 described above with respect to Figure 6 The system described is trained.

[0108] At step S802, the encoder model (e.g., encoder model 516) may receive training content to be used to generate a corresponding training latent space representation of the training content. The training content may be included in training data. The training content and training data are described in further detail above. In the example, the first training content includes pictures and / or videos and corresponds to the training prompt.

[0109] At step S804 , the decoder model may be used to generate output content using the latent space representation of the training content.

[0110] At step S806, the output content generated by the decoder model can be compared with the training content input to the encoder model (e.g., using the autoencoder adjustment system 608) to determine the degree of similarity between the two contents. The comparison can be performed using a loss function (e.g., reconstruction loss function, KL divergence loss, adversarial loss, perceptual loss). Additionally or alternatively, the comparison can be performed using a discriminator model, each of which has been described in further detail above. The discriminator model can be used when a watermark is added to the generated image.

[0111] KL divergence loss can be used with variational autoencoders (VAEs). KL divergence loss can be used to measure the difference between the learned latent distribution and the prior distribution (usually a standard Gaussian distribution). This forces the latent space to follow a specific distribution, thereby promoting better generation capabilities.

[0112] Adversarial loss can be used with Adversarial Autoencoders (AAEs), where a discriminator network is introduced along with the autoencoder. Adversarial loss is used to make the latent space distribution match the desired prior distribution. This may be similar to the loss used in Generative Adversarial Networks (GANs).

[0113] As an alternative to pixel-only reconstruction, the perceptual loss compares high-level features extracted from a pre-trained neural network (e.g., VGG) between the original and reconstructed images. This may be helpful for tasks such as super-resolution, where perceptual quality is more important than specific pixel-level accuracy.

[0114] At step S808, the weights of the encoder model and / or decoder model may be adjusted based on the comparison performed at step S806. The weight adjustment may be performed with the goal of minimizing a loss function or otherwise making the output content more similar to the training content.

[0115] Steps S802-S808 can be repeated for a certain number of training epochs to train or fine-tune the decoder model and / or encoder model. After the decoder model is trained, the trained decoder model can be used during inference time to generate content based on prompts (e.g., as part of the content generation system 108). The parameters of the encoder can be frozen after training and before being used during inference time. In some embodiments, after the encoder model is trained using the above process, the encoder model is used during the training process of the diffusion transformer model. The parameters of the encoder can be frozen after training and before being used during inference (e.g., when training the diffusion transformer model).

[0116] Figure 9An example of a process 900 for training a diffusion transformer model according to various embodiments of the present disclosure is shown. The diffusion transformer model may include a forward diffusion transformer (e.g., forward diffusion transformer 520) and a backward diffusion transformer (e.g., backward diffusion transformer 224). The backward diffusion transformer may be used by the content generation system (e.g., content generation system 108) during inference time.

[0117] At step S902, the training content may be received by an encoder model (e.g., encoder model 516), and the encoder model may generate potential training content for the training content. The training content may be included in the training data and may correspond to the training prompts. The training content, data, and prompts are described in further detail above.

[0118] At step S904 , the latent training content of the training content may be input to a diffusion-forward transformer (eg, a system or a model) so that the diffusion-forward transformer generates noise latent training content (eg, the noise latent training content 522 ).

[0119] At step S906, a training prompt (e.g., training prompt 502) corresponding to (e.g., describing attributes of the training content) the first training content may be received for use at steps S908 and / or S910. The training prompt may be included in a training data set. The training data set may be the same as the training data set used by the encoder model at step S902. In some embodiments, the training prompt describes the desired characteristics of the audio to be generated. For example, the desired characteristics may include style, color, theme, mood, texture, contrast, depth, motion, saturation, focus, perspective, narrative, and / or other characteristics. The training prompt may include audio, images, video, and / or sounds to be used to generate potential output content (e.g., potential output content 526). The training prompt may include content that the content generation system will add and / or modify.

[0120] At step S908, encoded hint conditions (e.g., encoded hint conditions 506) may be generated. The encoded hint conditions may be output from at least one hinted coding model. The encoded hint conditions may include a combination of outputs from at least one hinted coding model. The at least one encoded hint model may be included in a set of one or more hinted coding models (e.g., the second hinted coding model set 504). The encoded hint conditions may be transmitted to a backdiffusion transformer model (e.g., backdiffusion transformer 224) for use in training the backdiffusion transformer.

[0121] At step S910, a combined hint encoding (e.g., combined encoding 510) may be generated. The combined hint encoding may be generated using one or more coding models from a set of one or more hint encoding models (e.g., first hint encoding model set 508). The outputs of the coding models included in the set of one or more hint encoding models may be combined to generate the combined hint encoding. The combined hint encoding may be transmitted to a coding addition system to be added to the encoded time step. The encoded time step may be generated by encoding the time step using a time step coding model (e.g., time step coding model 204). The combined hint encoding is then added to the time step encoding to generate a conditional vector. The conditional vector may be used by an inverse diffusion transformer (e.g., inverse diffusion transformer 224) to generate potential output content.

[0122] At step S912, the back-diffusion transformer can generate potential output content using the noisy latent training content, the conditional vector, and the encoded prompt condition. The back-diffusion model can include one or more transformer blocks that operate in two separate domains to perform back-diffusion with separate weights in each domain, but also perform joint self-attention operations (e.g., QKV attention, multi-head attention) between the two domains. During the joint self-attention operation, the prompt embedding can be used to help the model focus on specific parts of the image embedding and further capture the contextual relationship between the prompt embedding and the prompt embedding.

[0123] At step S914, the noise potential training content generated at step S904, the first encoding hint condition generated at step S908, and / or the condition vector generated at step S912 can be input to the backdiffusion transformer. The backdiffusion transformer can use these inputs to generate potential output content (e.g., potential output content 526). The backdiffusion transformer can use these inputs to condition the performance of the backdiffusion transformer, as described above (e.g., with reference to FIG. Figures 2 to 5 The back-diffusion transformer can be configured to output potential output content in the conditioned latent space. The generated potential output content can represent the training content.

[0124] At step S916, the potential output content generated at step S914 can be compared to the potential training content generated at step S902. This comparison can be performed by a transformer adjustment system (e.g., transformer adjustment system 528). A loss function can be used to compare the potential training content and the potential output content.

[0125] At step S918, based on the comparison performed at step S916, a transformer weight adjustment signal can be transmitted to the forward diffusion transformer and / or the backward diffusion transformer (e.g., via a transformer adjustment system). Step 918 can be referred to as backpropagation. Step S918 can be performed in an effort to minimize the loss function used during the training process 900 and thereby improve the accuracy of the backward diffusion transformer.

[0126] Steps S902-S918 can be repeated for a certain number of training periods to train or fine-tune the decoder model and / or the encoder model. As training continues, the noisy potential training content generated by the forward diffusion transformer can be gradually more noisy (e.g., so that the noise becomes closer to representing a Gaussian distribution). As training continues, the backward diffusion transformer can become more accurate in correctly generating potential output content (similar to the potential training content) based on the noisy potential training content, temporal conditions, and first encoding hint conditions received as input. After the backward diffusion transformer is trained, the trained backward diffusion transformer can be used to generate content based on the hint during inference time (e.g., as part of the content generation system 108).

[0127] As a result of the training process, the content generation system can obtain a back-diffusion transformer that uses separate and corresponding weights for the two modalities and a bidirectional information flow between the tags for each modality to generate content with improved cue understanding and / or layout.

[0128] Figure 10 is a simplified block diagram illustrating an example architecture of a system 1000 for training and / or using the various models and systems described herein, in accordance with certain embodiments.

[0129] System 1000 includes computing system 104, network 1008, and server 1004. Computing system 104 can be similar to any user device and / or computing system described herein. Server 1004 can correspond to one or more server computers (e.g., a server cluster) of a cloud computing platform described herein.

[0130] The network 1008 may include any suitable communication path or channel, such as a wire or cable, optical fiber, a telephone line, a cellular connection, a radio frequency (RF) link, a WAN or LAN network, the Internet, or any other suitable medium. The network 1008 may include any one or a combination of various different types of networks, such as a cable network, the Internet, a wireless network, a cellular network, and other private and / or public networks. The network may use infrared, ultra-wideband (UWB), Bluetooth (BT), Bluetooth low energy (BTLE), Wi-Fi, and / or radio communication technologies.

[0131] Turning to further describe each element in detail, the computing system 104 can be any suitable computing device (e.g., a mobile phone, a tablet computer, a personal computer (PC), smart glasses, a smart watch, etc.). The computing system 104 has at least one memory 1010, one or more processing units (or processors) 1014, a storage unit 1016, a communication interface 1018, and an input / output (I / O) device 1020.

[0132] The processor(s) 1014 may be suitably implemented in hardware, computer-executable instructions, firmware, or a combination thereof. The computer-executable instructions or firmware implementation of the processor(s) 1014 may include computer-executable or machine-executable instructions written in any suitable computer language to perform the various functions described.

[0133] Memory 1010 can store program instructions that can be loaded onto and executed by processor(s) 1014, as well as data generated during the execution of these programs. Depending on the configuration and type of computing system 104, memory 1010 can be volatile (e.g., random access memory (RAM)) and / or non-volatile (e.g., read-only memory (ROM), flash memory, etc.). In some implementations, memory 1010 can include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), or ROM. Computing system 104 can also include additional storage 1016, such as removable or non-removable storage, including but not limited to magnetic storage, optical disks, and / or tape storage. Disk drives and their associated computer-readable media can provide non-volatile storage for computer-readable instructions, data structures, program modules, and other data for computing devices. In some embodiments, storage 1016 can be used to store audio, video, image, and / or text files.

[0134] The computing system 104 may also include a communication interface 1018 that allows the user device 1002 to communicate with servers, user terminals, and / or other devices over the network(s) 1008. The computing system 104 may also include one or more I / O devices 1020 to support, for example, connections to a keyboard, a mouse, a stylus, a voice input device, a touch screen input device, a display, speakers, a printer, and / or other components that the computing system 104 may include.

[0135] Turning to a more detailed description of the contents of memory 1010 , memory 1010 may include an operating system and one or more applications or services for implementing the features described herein, including the content generation system 108 or a system for training one or more models used in the content generation system 108 .

[0136] It should be understood that one or more functions of the content generation system 108 may be performed by the computing system 104 and / or the server 904 .

[0137] As described above, in some embodiments, remote server 1004 may correspond to a cloud computing platform. Remote server 1004 may perform one or more functions, including, for example: receiving a prompt; generating a first prompt code; generating a second prompt code; generating a first conditional latent space and a second conditional latent space based on the prompt code(s) and the first latent space; generating a third conditional latent space based on the first conditional latent space and the second conditional latent space; and / or generating content based on the third conditional latent space. Remote server 1004 may transmit the content to computing system 104. Remote server 1004 may include a credential generation module, I / O devices, and / or a communication interface.

[0138] Turning to a more detailed description of the contents of memory 1030 , memory 1030 may include an operating system 1032 and one or more applications or services for implementing the features described herein, including a communication module 1034 , an encryption module 1036 , a content generation system 108 , and / or a profile management module 1040 .

[0139] The communication module 1034 may include code that enables the processor 1046 to receive prompts, generate embeddings, train models, transmit content, and / or otherwise communicate with other system components. For example, the communication module 1034 may receive prompts and transmit content to the computing system 104.

[0140] The encryption module 1036 may include code that enables the processor 1046 to encrypt and / or decrypt messages. For example, the encryption module 1034 may receive encrypted data (e.g., prompts) from the computing system 104. The encryption module 1036 may include any suitable encryption algorithm for encrypting data. Suitable encryption algorithms may include the Data Encryption Standard (DES), Triple DES, the Advanced Encryption Standard (AES), and the like. The encryption module 1036 may also store (e.g., in the storage unit 1048) encryption keys (e.g., encryption and / or decryption keys) that can be used with such encryption algorithms. The encryption module 1036 may utilize symmetric or asymmetric encryption techniques to encrypt and / or authenticate data. For example, the computing system 104 may include code and / or keys similar to the encryption module 1036 that are suitable for encrypting / decrypting data communications with the computing system 104 (and / or server 1004).

[0141] The profile management module 1040 may include code that causes the processor 946 to maintain and store profiles for users and / or user devices. For example, the profile management module 940 may receive information about users and / or devices that are permitted to use the content generation system 108 and / or the training content generation system 108. The profile management module 1040 may keep track of users and / or devices associated with prompts and / or generated content so that when the user and / or device uses the server 1004 again, the prompts and / or generated content can be transmitted to the user and / or device (e.g., displayed as a content generation history). The profile management module 1040 may also include information about which users and / or user devices have which permissions, etc.

[0142] Figures 7 to 9 and any other drawings described in (and / or with respect to Figures 1 to 6 The processes described herein may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the corresponding system, implemented using hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., a storage device). The methods presented herein are intended to be illustrative and not limiting. Although Figures 7 to 9 The figures and other figures depict multiple process steps in a particular sequence or order, which is not intended to be limiting. In some alternative embodiments, the process may be performed in a different order or some steps may be performed in parallel. It should be understood that in alternative embodiments, Figures 7 to 9 The processes described in the other figures may include a greater or lesser number of steps than the processes described in the figures.

[0143] Each embodiment can further be implemented in various operating environments, and these operating environments can comprise one or more user computers, computing devices or processing equipment in some cases, and these devices can be used to operate any number of applications.User or client device can comprise any number of general personal computers, for example, desktop or laptop computers running standard operating systems, and cellular, wireless and handheld devices running mobile software and capable of supporting multiple networking and message transfer protocols.Such system can also comprise multiple workstations, runs multiple commercially available operating systems and such as any one in other known applications for development and data management purposes.These devices can also comprise other electronic devices, such as virtual terminals, thin clients, gaming systems and other devices that can communicate via a network.

[0144] Most embodiments utilize at least one network well known to those skilled in the art to support communications using any of a variety of commercially available protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), Open Systems Interconnection (OSI), File Transfer Protocol (FTP), Universal Plug and Play (UPnP), Network File System (NFS), Internet File Sharing System (CIFS), and AppleTalk. The network can be, for example, a local area network, a wide area network, a virtual private network, the Internet, an intranet, an extranet, a public switched telephone network, an infrared network, a wireless network, or any combination thereof.

[0145] In embodiments utilizing a web server, the web server may run any of a variety of server or mid-tier applications, including a Hypertext Transfer Protocol (HTTP) server, an FTP server, a Common Gateway Interface (CGI), a data server, a JAVA server, and a commercial application server. The server(s) may also execute programs or scripts in response to requests from user devices, for example by executing one or more web applications, which may be implemented in any programming language (such as One or more scripts or programs written in C, C#, or C++) or a scripting language (e.g., Perl, Python, or TCL), and combinations thereof. The server(s) may also include a database server, including but not limited to and Commercially available database servers.

[0146] The environment may include a variety of data storage and other memories and storage media, as discussed above. They may reside in a variety of locations, such as on a storage medium local to (and / or therein) one or more computers, or remotely from any or all computers via a network. In a particular embodiment set, information may reside on a storage area network (SAN) well known to those skilled in the art. Similarly, any necessary files for performing the functions attributed to a computer, server, or other network device may be stored locally and / or remotely where appropriate. When a system includes a computerized device, each such device may include hardware elements that may be electrically coupled via a bus, including, for example, at least one central processing unit (CPU), at least one input device (such as a mouse, keyboard, controller, touch screen, or keypad), and at least one output device (such as a display device, printer, or speaker). Such a system may also include one or more storage devices, such as disk drives, optical storage devices, and solid-state storage devices, such as random access memory (RAM) or read-only memory (ROM), and removable media devices, memory cards, flash memory cards, etc.

[0147] Such devices may also include a computer-readable storage medium reader, a communication device (e.g., a modem, a network card (wireless or wired), an infrared communication device, etc.), and the above-mentioned working memory. The computer-readable storage medium reader may be connected to or configured to receive a computer-readable storage medium, which represents a remote, local, fixed and / or removable storage device, and a storage medium for temporarily and / or more permanently containing, storing, transmitting, and obtaining computer-readable information. The system and numerous devices also typically include multiple software applications, modules, services, or other elements located in at least one working memory device, including an operating system and application programs, such as a client application or a web browser. It will be understood that alternative embodiments may have a large number of changes compared to the above-described embodiments. For example, customized hardware may also be used, and / or specific elements may be implemented in hardware, software (including portable software, such as applets), or both. Further, connections to other computing devices such as network input / output devices may also be used.

[0148] Storage media computer-readable media for containing the program / code, or a portion of the program / code, may include any suitable media known or used in the current art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing and / or transmitting information (such as computer-readable instructions, data structures, program modules or other data), including RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other storage technology, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the required information and can be accessed by system devices. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will recognize other ways and / or methods for implementing various embodiments.

[0149] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope as set forth in the claims.

[0150] Other variations are within the spirit of the present disclosure. Thus, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrative embodiments thereof have been shown in the drawings and described above in detail. However, it should be understood that there is no intent to limit the present disclosure to the particular form or forms disclosed, but on the contrary, the intent is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the present disclosure, as defined in the appended claims.

[0151] When describing the disclosed embodiments (especially in the context of the following claims), the terms "a," "an," and "the" and similar referents should be construed to include both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "comprising" should be construed as open-ended terms (i.e., "including but not limited to") unless otherwise indicated. The term "connected" should be interpreted as partially or completely contained within, attached to, or connected together, even if there are intervening substances. The numerical ranges recited herein are merely intended as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is included in this specification as if it were recited individually. All methods described herein can be performed in any suitable order unless otherwise indicated or clearly contradicted by context. Any examples or illustrative language (e.g., "such as") provided herein are intended merely to better illustrate the embodiments of the present disclosure and do not limit the scope of the disclosure unless otherwise indicated. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0152] Disjunctive language such as "at least one of X, Y, or Z" should be understood in the context of common usage, unless otherwise specifically stated, to mean that something, a term, etc., can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain embodiments require at least one of X, at least one of Y, or at least one of Z to be present.

[0153] Preferred embodiments of the present disclosure are described herein, including the best mode of practicing the present disclosure known to the inventor. By reading the foregoing description, variations of these preferred embodiments may be apparent to those of ordinary skill in the art. The inventors anticipate that skilled artisans will adopt these variations as needed, and the inventors intend that the present disclosure be practiced in a manner different from that specifically described herein. Therefore, the present disclosure includes all modifications and equivalents of the subject matter described in the appended claims as permitted by applicable law. In addition, unless otherwise indicated herein or clearly contradicted by context, the present disclosure includes any combination of the above-mentioned elements in all possible variations thereof.

[0154] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

Claims

1. A system comprising: one or more storage media storing instructions; as well as One or more processors configured to execute the instructions so that the system: receiving a prompt describing desired characteristics of the image; generating a prompt encoding based on the prompt using a set of encoding models; generating, using a first transformer block in a diffusion transformer model, a first cue embedding and a first image embedding based on the cue encoding and a noise input; generating a second image embedding based on the first image embedding and the first cue embedding using a second transformer block in the diffusion transformer model; as well as Based on the second image embedding, the image is generated.

2. The system of claim 1, wherein the set of coding models comprises: First encoding model subset; as well as A second coding model subset different from the first coding model subset. 3 . The system of claim 2 , wherein the first subset of coding models comprises a different number of coding models than the second subset of coding models.

4. The system of claim 1, wherein the set of encoding models comprises at least one of: a first text encoder trained jointly with an image encoder, or a second text encoder trained as a text-to-text encoder.

5. The system of claim 1 , wherein the diffusion transformer model is trained using cue embeddings generated by a second set of encoding models, the second set of encoding models being different from the set of encoding models.

6. The system of claim 1 , wherein generating the first hint embedding is further based on a first weight included in a first weight set associated with a first domain of the first hint embedding, and generating the first image embedding is further based on a second weight included in a second weight set associated with a second domain of the first image embedding.

7. The system of claim 6, wherein the diffusion transformer model is trained based on the instructions, the instructions further causing the system to: Generate a noisy latent input using the input; generating a potential output content using the noisy potential input content and a third hint encoding corresponding to the noisy potential input content, and At least one of the first weight or the second weight is adjusted based on comparing the noise potential input content and the potential output content.

8. The system of claim 1, wherein generating the first image embedding or the first hint embedding comprises applying an attention operation to intermediate values ​​generated based on at least one of the noise input or the hint encoding.

9. A computer-implemented method comprising: receiving a prompt describing desired characteristics of the image; generating a prompt encoding based on the prompt using a set of encoding models; generating, using a first transformer block in a diffusion transformer model, a first cue embedding and a first image embedding based on the cue encoding and a noise input; generating a second image embedding based on the first image embedding and the first cue embedding using a second transformer block in the diffusion transformer model; as well as Based on the second image embedding, the image is generated.

10. The computer-implemented method of claim 9, wherein generating the first hint embedding further comprises: generating a first vector space using the hint encoding and the encoding of the time steps; generating a second vector space using the hint encoding; as well as generating the first prompt embedding using the first vector space, the second vector space, and the noise input; and Wherein generating the first image embedding further comprises: Generate the noise input using position encoding and noise pixel encoding; The first image embedding is generated using the first vector space, the second vector space, and the noise input. The computer-implemented method of claim 10 , wherein the time step is given a greater weight when the time step is an intermediate time step.

12. The computer-implemented method of claim 9, wherein generating the first hint embedding further comprises: Using more than one encoding model included in the set of encoding models, textual conditioning is generated that includes encodings from at least two encoding models.

13. The computer-implemented method of claim 9, wherein generating the first hint embedding and the first image embedding comprises: generating a first normalized median value of the first prompt embedding; generating a second normalized intermediate value of the first image embedding; combining the first normalized intermediate value and the second normalized intermediate value; as well as A self-attention operation is performed on the joint value.

14. The computer-implemented method of claim 13, wherein the hint code is a first hint code, and generating the first hint code comprises: generating a second hint encoding using the hint as input to a subset of encoding models included in the set of encoding models; generating a third hint encoding using the hint as input to a second subset of encoding models included in the set of encoding models; as well as The first hint code is generated by combining the second hint code and a portion of the third hint code.

15. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors of a system, cause the system to perform operations comprising: receiving a prompt describing desired characteristics of the image; generating a prompt encoding based on the prompt using a set of encoding models; generating, using a first transformer block in a diffusion transformer model, a first cue embedding and a first image embedding based on the cue encoding and a noise input; generating a second image embedding based on the first image embedding and the first cue embedding using a second transformer block in the diffusion transformer model; as well as Based on the second image embedding, the image is generated.

16. The non-transitory computer-readable storage medium of claim 15, wherein training the diffusion transformer model comprises: pre-training the diffusion transformer model using images of a first size; as well as The diffusion transformer model is fine-tuned using at least an image having a second size larger than the first size, or an image having a mixed aspect ratio.

17. The non-transitory computer-readable storage medium of claim 15, wherein the first hint embedding and the first image embedding have a common dimension.

18. The non-transitory computer-readable storage medium of claim 15, wherein the first transformer block and the second transformer block each comprise a linear layer trained using a learnable low-rank (LoRA) matrix.

19. The non-transitory computer-readable storage medium of claim 15, wherein the diffusion transformer model is trained using training data comprising an image set and a prompt set, the prompt set comprising synthetic prompts generated using corresponding images from the image set.

20. The non-transitory computer-readable storage medium of claim 15, wherein the diffusion transformer model comprises a rectified flow model and is trained using a logit-normal distribution.

Images