Content synthesis using a generative model of artificial intelligence

Abstract

A system that includes the following: one or more storage media that store instructions; and one or more processors configured to execute the instructions to cause the system to: to receive a request that describes a desired property of an image; to generate a prompt coding based on the prompt using a set of coding models; to generate a first prompt embedding and a first image embedding based on the prompt encoding and a noise input using a first transformer block of a diffusion transformer model; using a second transformer block of the diffusion transformer model, to create a second image embedding based on the first image embedding and the first prompt embedding; and to generate the image based on the second image embedding.

Description

CROSS-REFERENCES TO RELATED REGISTRATIONS

[0001] This application is for a German utility model claiming priority from EP application no. 25159213.5, which in turn claims priority from US preliminary application no. 63 / 567,127 filed on 19 March 2024 and US preliminary application no. 63 / 633,020 filed on 11 April 2024, the contents of which are incorporated herein by reference in full for all purposes. BACKGROUND

[0002] Artificial intelligence (AI) models (e.g., machine learning (ML) models) can be used to generate output based on received natural language prompts. Some AI models can be used to generate and output content (e.g., images) based on natural language prompts. For example, a machine learning model might receive a user prompt asking the model to "generate a picture of a cat sleeping on a blanket." The machine learning model could then generate an image showing a cat sleeping on a blanket. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Various embodiments in accordance with the present disclosure are described with reference to the drawings, in which: Fig. Figure 1 shows an example of the use of a system for generating content according to embodiments of the present disclosure; Fig. Figure 2 shows an example of a system for generating content according to embodiments of the present disclosure; Fig. Figure 3 shows an example of a transformer block of a diffusion transformer model according to embodiments of the present disclosure; Fig. Figure 4A shows an exemplary system for generating content according to embodiments of the present disclosure; Fig. Figure 4B shows an exemplary block contained in a reverse diffusion transformer according to embodiments of the present disclosure; Fig. Figure 5 shows an example of a system for training a diffusion transformer according to embodiments of the present disclosure; Fig. Figure 6 shows an example of a system for training a coding model and / or a decoding model according to embodiments of the present disclosure; Fig. Figure 7 shows an example of a method for using a content creation system according to embodiments of the present disclosure; Fig. Figure 8 shows an example of a method for training a coding model and / or a decoding model according to embodiments of the present disclosure; Fig. Figure 9 illustrates an example of a method for training a diffusion transformer model according to embodiments of the present disclosure; and Fig. Figure 10 is a simplified block diagram illustrating an example architecture of a system used for training and / or using the models and systems described herein, according to some embodiments. DETAILED DESCRIPTION

[0004] The following description details various embodiments. For explanatory purposes, specific configurations and details are presented to enable a comprehensive understanding of the embodiments. However, it is obvious to those skilled in the art that the embodiments can also be implemented without these specific details. Furthermore, known features may be omitted or simplified so as not to obscure the described embodiment.

[0005] There are challenges related to machine learning (ML) models that generate content in response to prompts. Improvements can be made to what a machine learning model can generate (e.g., images), how the machine learning model is trained, and how the input can influence the generated content output. For example, improvements can enable content generation in a scalable, efficient, high-quality, and accurate manner.

[0006] The use of transformer models in generative models for content (e.g., images) has been limited to date. This tendency is reflected in a general conventional preference for a fully convolutional neural network architecture (e.g., U-Net) in diffusion models. The inductive bias of a fully convolutional neural network does not necessarily make it the best choice for diffusion models. The transformer models presented here can be used instead of a fully convolutional neural network architecture for content generation (e.g., text-to-image generation).

[0007] Embodiments of the present disclosure relate to techniques for generating content based on a request. The request may contain text, video, audio, and / or an image. The generated content may include an image, a video, an image with a specific style, an image with a specific resolution, an image with a specific aspect ratio, and / or content with one or more other properties. Latent diffusion models can be used to generate content with high-resolution synthesis. Embodiments can offer improvements over conventional systems by enabling the generation of content on a large scale, efficiently (e.g., with efficient use of resources such as processing resources, memory resources, and network resources), and in high quality, making these embodiments useful for generative modeling.

[0008] The embodiments described herein can construct a sequence comprising encodings of two modalities (e.g., text input and image input). The sequence can include positional encodings and flattened patches of a latent pixel representation. After encoding and concatenating the patch encoding and the text encoding onto a common dimensionality, the disclosed reverse diffusion transformers can apply a sequence of modulated attention and multi-layer perceptrons (MLPs) to generate content based on the text input (e.g., a prompt). Given the conceptual differences between two distinct modalities (e.g., text and image encodings), embodiments can employ separate sets of weights for each modality. While an independent transformer is used for each modality, the embodiment can apply the sequences of each modality to an attention operation (e.g.,combined (shared self-awareness) so that both representations can work in their respective areas while taking each other into account.

[0009] The exposed latent diffusion models, which use separate weights for each modality (e.g., text modality and image modality) and / or are configured for bidirectional information flow, can be referred to as a multimodal latent diffusion model (MMDiT). The MMDiT can improve text comprehension and spelling skills compared to traditional techniques. For example, it can more accurately reproduce text in generated images and ensure that text elements such as fonts, styles, and sizes are displayed correctly. Furthermore, the MMDiT architecture enables the efficient and effective generation of high-quality images based on text input.

[0010] The implementations can include a rectified flow (RF) formulation, which combines data and noise along a linear path during training. The inclusion of RF can lead to more straightforward inference paths, enabling sampling with fewer steps and thus in less time and using fewer resources (e.g., processing resources, network resources). Rectified flow can be used to train a diffusion model.

[0011] Implementations can improve memory utilization by removing one or more memory-intensive coding models used during training from the inference phase. Removing one or more coding models can significantly reduce memory requirements with minimal performance loss.

[0012] Implementations can effectively handle multi-subject prompts that contain detailed descriptions of scenes, compositions, and / or scenarios with more than one object, person, or concept. Multi-subject prompts provide rich and complex information from which implements can generate appropriate content (e.g., images) that accurately depicts the described scene or scenario. Effective handling of multi-subject prompts requires the devices (e.g., the MMDiT) to understand and interpret the relationships between the various subjects mentioned in the prompt in order to produce coherent and realistic images.Effective handling of prompts increases the likelihood of generating the desired content with minimal deviation from the intended concept or scene, thereby reducing resource consumption during creation compared to previous techniques. For example, if content creation is correct after one prompt instead of two, fewer network, storage, processing, and / or energy resources can be consumed.

[0013] Fig. Figure 1 shows an example of the use of a content generation system 108 according to embodiments of the present disclosure. The content generation system 108 can be used as part of a content generation system 100. The content generation system 100 can comprise a computer system 104, a network 106, and the content generation system 108. The content generation system 108 can receive a request (e.g., a natural language request) from the computer system 104, which causes the content to be generated using one or more machine learning (ML) models 110. The generated content can be transferred to the computer system 104 and presented via a user interface.

[0014] The computer system 104 can be a user device (e.g., laptops, personal computers, telephones, etc.). The computer system 104 can be a server. The computer system 104 can be capable of receiving input from a user 102, e.g., via a user interface. In certain embodiments, the input received by the computer system 104 includes the prompt. The input can cause the computer system 104 to transmit the prompt to the content generation system 108 (e.g., via the network 106). For example, a user interface of the computer system 104 can receive a natural language prompt (e.g., from the user 102) describing the desired properties of the content to be included in the generated content, and the natural language prompt can be transmitted to the content generation system 108 via the network 106.

[0015] The request can contain text (e.g., natural language text) that describes the desired characteristics of the content to be generated, such as one or more images and / or one or more videos. These characteristics can describe a style, color, theme, mood, texture, contrast, depth, movement, saturation, focus, perspective, narrative, and / or other feature to be included in the generated content. The request can contain at least one text, one audio, one image, and / or one video. In some embodiments, the text can describe a scene (e.g., a scene from a book or screenplay) that can then be used to generate content that corresponds to the text.In some embodiments, audio, image(s), and / or video(s) can be included in the prompt to cause the Content Creation System 108 to generate content that corresponds to the audio, image(s), and / or video(s). For example, a portion of an image can be included in the prompt, and content can be generated that includes that portion or similar characteristics to that portion. In another example, a video scene from a movie can be included in the prompt, and the System 108 can generate content that has similar characteristics (e.g., similar style, colors, themes, mood, texture, contrast, depth, motion, saturation, focus, perspective, narrative, etc.) to the section.

[0016] The request or other information from Computer System 104 may contain information to determine one or more encoders to use. In one example, the encoders used to encode the request may be predetermined and remain constant during runtime. In another example, the request may explicitly specify which encoders or set of encoders to use. In yet another example, the information contained in the request may be used by Content Creation System 108 to determine one or more encoders and / or one or more sets of encoders to use to encode the request or a portion thereof.

[0017] The prompt can be used as input for the content creation system 108 to initiate content generation. The content creation system 108 can use a set of one or more machine learning models 110 to create the content using the prompt. The set of one or more machine learning models 110 can include one or more encoding models, a decoding model, and / or a latent diffusion model (e.g., a diffusion transformer model). The training and use of such models are described in detail here.

[0018] The generated content can contain features defined by the request. The content can include an image or a video. The generated content can have one or more predefined properties. For example, the content can have a predefined size (e.g., pixel dimensions, number of pixels, bit size) and a predefined maximum size. The content generation system 108 can transfer the generated content to the computer system 104 for presentation (e.g., for display, for presentation as a downloadable file).

[0019] By using computer system 104 to present the content to user 102, user 102 can view the content. Computer system 104 can store the content in memory and send the content to another computer system (e.g., a social media application, another user device, etc.). In some embodiments, subsequent requests (e.g., from computer system 104 or another computer system) can be received by content generation system 108 to cause content generation system 108 to modify the generated content.

[0020] Network 106 can be configured to connect computer system 104 and content generation system 108, as shown. Network 106 can be configured to connect any combination of the system components. In certain embodiments, network 106 is not part of content generation system 100. For example, content generation system 108 can run locally on computer system 104 and / or one or more of the machine learning models 110 can run locally on computer system 104.

[0021] Each of the data connections of the network 106 can be implemented via a public (e.g., the Internet) or private network (e.g., an intranet), whereby an access point, a router, and / or another network node can communicatively couple the computer system 104 and the content creation system 108. A data connection between the components can be a wired data connection (e.g., a Universal Serial Bus (USB) connection) or a wireless connection (e.g., a radio frequency-based connection). Data connections can also be established via a mesh network. A data connection can also represent a power connection. A power connection can supply power to the connected component. The data connection can provide data transfer to and from the system components. Professionals know that devices connected via a network (e.g., a local area network (LAN), a wide area network (WAN), etc.)) can be communicatively coupled. Further devices can be communicatively coupled through a combination of wired and wireless means (e.g., a wireless connection to a router that is connected to a server via an Ethernet cable).

[0022] The interfaces between components that communicate with the content creation system (100), as well as the interfaces between components within the content creation system (100), can be implemented via web interfaces and / or application programming interfaces (APIs). For example, computer system 104 can implement a number of APIs for communication with the content creation system 108 and / or user interfaces of computer system 104. In one example, computer system 104 uses a web browser while communicating with the content creation system 108.

[0023] The in Fig. The depicted content creation system 100 can further implement the depicted steps S120-S126. The depicted steps can be implemented by executing instructions stored in a memory of the content creation system 100, the execution being carried out by processors of the content creation system 100.

[0024] In step S120, a request can be transmitted from computer system 104 to network 106. The request can contain information received from a user interface of computer system 104. For example, user 102 may have entered: "Please create an image of an old, rusty robot wearing trousers and a jacket and skiing in a supermarket," and the request can then output the entered information and transmit it to network 106.

[0025] In step S122, the command prompt can still be transmitted from computer system 104 to content creation system 108 via network 106. After content creation system 108 receives the command prompt, it can use one or more machine learning models 110 to create the content using the command prompt.

[0026] In step S124, the content generation system 108 can transfer the generated content to the network 106.

[0027] In step S126, network 106 can transmit the generated content to computer system 104. After computer system 104 receives the generated content, it can present the generated content, or parts of it, via its user interface. For example, computer system 104 can present an image or video on a display that can be viewed by user 102.

[0028] Fig. Figure 2 shows an example of a content generation system 108 according to embodiments of the present disclosure. The content generation system 108 can be the one described in Fig. The content generation system 108 described in section 1 can be configured to receive a prompt 208 and output generated content 230. The content generation system 108 can include one or more encoding models, an encoding addition system 214, a reverse diffusion transformer 224, and a decoding model 228. The one or more encoding models can include one or more prompt encoding models and a time-step encoding model 204.

[0029] The prompt 208 can be transmitted by a computer system (e.g., the computer system 104 described above). The prompt 208 can be received by a system (e.g., over a network). The prompt 208 can be received by a user interface of the system. The prompt 208 can describe the desired characteristic of the content to be generated by the content creation system 108. For example, a size (e.g., pixel dimensions, pixel count, bit size), a style, a color, a theme, a mood, a texture, a contrast, a depth, a motion, a saturation, a focus, a perspective, a narrative. The prompt 208 can be received by one or more prompt encoding models.

[0030] A prompt encoding model within the set of prompt encoding models can be configured to represent prompt 208, or a portion thereof, in a multidimensional space (e.g., a vector space). The prompt encoding model can include neural network layers to convert prompt 208, or a portion thereof, into a prompt encoding (e.g., first prompt encoding(s) 212, prompt conditioning 220) in the high-dimensional space. The neural network layers used to generate the prompt encoding can be referred to as embedding layers. The prompt encoding model can be configured and / or pre-trained to generate encodings for prompts represented as text, audio, image, and / or video. The prompt encoding model can be a combined image and text encoding model (e.g.,a Contrastive Language-Image Pre-Training (CLIP) model, a text encoder from a CLIP model, a large language model, a T5 model, a Convolutional Neural Network Transformer, or a recurrent neural network. A person skilled in the art with knowledge of the present disclosure would recognize other ML models that can be used for prompt coding.

[0031] The prompt coding models can comprise a first set of prompt coding models 210 and / or a second set of prompt coding models 218. A set of prompt coding models can contain one or more prompt coding models. The prompt coding models can contain one or more frozen prompt coding models (e.g., trainable model attributes are preserved). The set of prompt coding models used to encode Prompt 208, or a portion thereof, can be determined based on Prompt 208. For example, the set of prompt coding models used to encode Prompt 208, or a portion thereof, can be determined based on instructions in Prompt 208 (e.g., to use a specific set of prompt coding models).In one example, the set of prompt encoding models used to encode Prompt 208, or a portion thereof, can be determined based on information contained within Prompt 208 (e.g., Prompt contains text, Prompt contains text and image, Prompt contains audio, etc.). The set of prompt encoding models used to encode Prompt 208, or a portion thereof, can be predefined (e.g., by a system administrator). Alternatively, the set of prompt encoding models used to encode Prompt 208, or a portion thereof, can be determined based on instructions received from a computer system.

[0032] In some embodiments, the prompt 208, or a part of the prompt 208, is received by the first set of prompt coding models 210 and / or the second set of prompt coding models 218. The first set of prompt coding models 210 may contain one or more prompt coding models to generate an encoding of at least a part of the prompt 208. The encodings generated by the first set of prompt coding models 210 may be combined (e.g., by concatenation) into a single vector space, represented by the combined encoding 212. The first set of prompt coding models 210 and the second set of prompt coding models 218 may contain one or more of the same prompt coding models (e.g., a common CLIP model).The first set of prompt coding models 210 and the second set of prompt coding models 218 can contain a different number of prompt coding models. The second set of prompt coding models 218 can contain one or more prompt coding models to generate an encoding of at least a part of the prompt 208. The generated encodings from the second set of prompt coding models 218 can be combined (e.g., by concatenation) into a single vector space represented by prompt conditioning 220. The vector space of prompt conditioning 220 can have a dimensionality corresponding to the dimensionality of a noisy latent space 222 that is input into the reverse diffusion transformer 224.

[0033] The time-step coding model 204 can be used to encode time step 202 as encoded time step 206. A time step 202 can be received by the time-step coding model 204. Time step 202 can represent a time step of the reverse diffusion process. The time-step coding model 204 can encode time step 202 using a neural network and / or encode time step 202 based on a function. For example, the time-step coding model 204 can use a sine function to determine the encoded time step 206 based on time step 202. The output of the sine function can be represented in a vector space as encoded time step 206. The vector space of the encoded time step 206 can have the same dimensionality as the combined encodings 212.

[0034] The coding addition system 214 can add the vector of the encoded time step 206 with the vector of combined encodings 212 to generate the time condition 216. The time condition 216 can be used by a modulation attention mechanism of the reverse diffusion transformer 224 and can enable conditional generation. The time condition 216 can be given a higher weight if the time step 202 used to generate the time condition 216 is closer to the center of a time window than other time steps that are farther from the center (e.g., an intermediate step).

[0035] The reverse diffusion transformer 224 can receive time conditioning 216, prompt conditioning 220, and noisy latent space 222 as input. The reverse diffusion transformer 224 can use the inputs to generate a conditioned latent space 226. The noisy latent space 222 can be a latent space containing randomly generated noise. The noisy latent space 222 can be generated based on sample values ​​according to a distribution (e.g., a Gaussian distribution). The noisy latent space 222 can be generated based on a seed. The seed can be input into the content creation system 108 (e.g., via a user interface). The noisy latent space 222 can be stored in memory and used by the reverse diffusion transformer 224.

[0036] The noiseless latent space 222 can contain positional information. In some embodiments, the noisy latent space 222 is generated by adding a positional embedding to an initial noisy latent space. The initial noisy latent space may have been generated using the techniques described above with respect to the noisy latent space 222. The initially noisy latent space may represent a pixel encoding. The positional embedding can add information about the position of elements in the noisy latent space 222. The positional embedding can help the reverse diffusion transformer understand relative positions and relationships between different parts of an image.

[0037] The Reverse Diffusion Transformer 224 can be a machine learning model trained to generate a conditioned latent space (e.g., Conditioned Latent Space 226) using a noisy latent space (e.g., Noisy Latent Space 222). Techniques for training the Reverse Diffusion Transformer 224 are described in detail here. The Conditioned Latent Space 226 can be generated using a combination of Prompt Conditioning 220, Temporal Conditioning 216, and Noisy Latent Space 222.

[0038] The reverse diffusion transformer 224 can generate a conditioned latent space 226 by removing noise from the noisy latent space 222. The reverse diffusion transformer 224 can iteratively remove noise from the noisy latent space 222 over time steps (e.g., time step 202) to obtain the conditioned latent space 226. The reverse diffusion transformer 224 can use one or more transformer blocks, which are described in more detail below, to generate the conditioned latent space 226. The conditioned latent space 226 can be viewed as an encoded form of the content (e.g., the generated content 230). The conditioned latent space 226 can be stored in the memory of the content creation system 108.

[0039] The decoding model 228 can receive the conditioned latent space 226 as input and use the conditioned latent space 226 to generate the content 230. The decoding model 228 can be trained using the techniques described herein. The decoding model 228 can be configured to receive the conditioned latent space 226 after it has been output by the reverse diffusion transformer 224. The decoding model 228 can include neural network layers that are used to generate content from an encoding of content (e.g., the conditioned latent space 226). The decoding model 228 can comprise a recurrent neural network, a long-term memory network, a transformer model, a convolutional neural network, or another model architecture.A person skilled in the art who is familiar with the advantages of the present disclosure would recognize other architectures that could be used for the decoding model 228.

[0040] Fig. Figure 3 shows an example of a transformer block 300 of a diffusion transformer model (e.g., reverse diffusion transformer 224) according to embodiments of the present disclosure. Transformer block 300 can be one of several (e.g., 15, 38, many) transformer blocks included in the diffusion transformer model. Transformer block 300 can receive input signals from other transformer blocks and / or send output signals to other transformer blocks.

[0041] Transformer block 300 can receive prompt conditioning 220, latent noise space 222, and time conditioning 216 as input (e.g., prompt conditioning 220, latent noise space 222, and time conditioning 216 as described above). Transformer block 300 can receive input from a previous transformer block if it is not the first transformer block of the reverse diffusion transformer. Transformer block 300 can receive input from a coding addition system (e.g., coding addition system 214), a second set of prompt coding models (e.g., second set of prompt coding models 218), and a noisy latent space generation system. The transformer block 300 can use the inputs to create a conditioned prompt 320 and / or a conditioned latent space 322.The conditioned latent space can be the conditioned latent space 226 described above. Transformer block 300 can transfer the conditioned prompt 320 and the conditioned latent space 322 to a subsequent transformer block. Transformer block 300 can transfer the conditioned latent space 322 to a decoding model (e.g., decoding model 228).

[0042] The transformer block 300 can operate separately on the prompt conditioning vector 220 and the noisy latent space vector 222 for some of the operations performed. A sequence of operations can be executed separately on the prompt conditioning vector 220 and the noisy latent space vector 222 because the vectors can represent encodings that contain many conceptual differences (e.g., an image encoding and a text encoding). For example, the prompt conditioning 220 can be subjected to layer normalization (layer norm) operations 302a, modulation 304a, and / or linear operations 306a to generate an initial prompt intermediate value 324.

[0043] Layer normalization 302a can be used to ensure that neurons in a shared layer have the same normalization term (e.g., the same mean and variance). By normalizing the distributions of intermediate layers, layer normalization 302a can enable smoother gradients, faster training, and greater accuracy. Adaptive layer normalization (adaLN) can be used to condition the diffusion network to text representations, allowing for parameter-efficient fitting.

[0044] The modulation mechanism can enable conditional generation. The modulation can utilize time conditioning (216). The modulation mechanism can employ scaling (e.g., adjusting a data range) and shift operations (e.g., shifting a data distribution). Scaling and shift operations can make data features more suitable for modeling.

[0045] Linear 306a operations can perform linear transformations on the input. They can be used to clean data, extract features, and / or prepare data for further operations. Linear 306a layer operations can result from training with machine learning low-rank matrices (LoRA).

[0046] Similar to layer normalization 302a, modulation 304a, and linear 306a, the layer normalization 302, modulation 304b, and / or linear 306b operations can be performed on the noisy latent space to generate a first noisy intermediate value 326. After separately processing the prompt conditioning vector 220 and the noisy latent space vector 222, the first intermediate value 324 for prompt conditioning and the first intermediate value 326 for noise generated by the respective operations can be combined (e.g., by concatenation). The combination can be performed by a concatenation system 308 configured to concatenate two vectors.

[0047] The combined vectors can be used by a joint self-attention system 310 to generate an output. The joint self-attention system 310 combines the sequences of the first intermediate prompt value 324 and the first noisy intermediate value 326 (e.g., of different modalities) for the attentional operation, so that both representations can operate in their respective vector spaces while taking each other into account. The joint self-attention system 310 can enable the capture of contextual relationships between the two intermediate embedding spaces. The output of the joint self-attention system 310 can be processed with two separate sequences of operations. For example, each sequence of operations can include any combination of a linear transformation, layer normalization, modulation, coding addition, and / or the use of a multilayer perceptron (MLP).An MLP can be configured to perform multiple layers of nonlinear transformations on the input. An exemplary first sequence of operations performed on the output of the joint self-attention system 310 is shown as linear operations 312a, coding addition using the coding addition system 214, coding addition using the coding addition system 214, layer normalization 314a, modulation 316a, MLP 318a, and coding addition using the coding addition system 214. An exemplary second sequence of operations performed on the output of the joint self-attention system 310 is shown as linear operations 312a, coding addition using the coding addition system 214, layer normalization 314a, modulation 316a, MLP 318a and coding addition using the coding addition system 214.

[0048] After the output of the shared self-attention system 310 has been processed using the two separate operation sequences, the conditioned prompt 320 and the conditioned latent space 322, generated by the respective operation sequence, can be output by block 300. The sequence of separate operations can correspond to a transformer with independent weights for each modality, but allows each of the two transformers to perform operations informed by the other.

[0049] As an example, the conditioned prompt 320 can be generated from a common self-attention system, an initial sequence of operations (e.g., 302a, 304a, 306a, 308, 312a, 214, 314a, 316a, and / or 318a), and at least one initial weight contained in an initial set of weights associated with an initial domain (e.g., a text domain) of the conditioned prompt 320. The conditioned latent space 322 can be generated from a common self-attention system, a second sequence of operations (e.g., 302b, 304b, 306b, 308, 312b, 214, 314b, 316b, and / or 318b), and at least one second weight contained in a second set of weights assigned to a second domain (e.g., a content domain) of the conditioned latent space 322.

[0050] Fig. Figure 4A shows an exemplary content generation system 400 according to embodiments of the present disclosure. Fig. Figure 4B shows an exemplary transformer block 414 (e.g., 414a, 414b to 414d) contained in the reverse diffusion transformer 224, according to embodiments of the present disclosure. The exemplary transformer block 414 is an example of transformer block 300. Identical components can be identified by the same part numbers. For example, the linear 404b operations can be similar to the linear 404a operations.

[0051] An exemplary content generation system 400 can receive a prompt 208 (e.g., as described above) and generate content 230 (e.g., as described above) based on the prompt 208. The exemplary content generation system 400 comprises a first set of prompt encoding models 210, a second set of prompt encoding models 218, a time-step encoding model 204, a reverse diffusion transformer 224, and a decoding model 228, each with respect to at least Fig. 2 have been described.

[0052] The first set of prompt coding models 210 can generate one or more encodings of the prompt 208. In the illustrated example, the first set of prompt coding models 210 includes a CLIP-G / 14 coding model and a CLIP-L / 14 coding model. In certain embodiments, more or fewer coding models are used. In certain embodiments, a different combination of coding models is used. The first set of prompt coding models 210 can include one or more coding models that were used during the training of the exemplary content creation system 400. In the illustrated embodiments, each of the coding models included in the first group of prompt coding models 210 generates an encoding of at least part of the prompt 208.

[0053] Exemplary content creation system 400 shows an embodiment in which the encoding outputs of the CLIP-G / 14 encoding model and the CLIP-L / 14 encoding model can be combined and have sizes of 768 and 1280, respectively. The combined outputs can be concatenated to perform vector conditioning c. vec ∈ ℝ 2048 to obtain. c vec is represented as combined code 212. Combined code 212 may contain coarse-grained information about request 208.

[0054] The penultimate hidden representations of the CLIP-G / 14 coding model and the CLIP-L / 14 coding model can be combined channel-wise into a CLIP- cctxtCLIP∈ℝ77×2048 They can be chained together. c can then become a final hidden representation. cctxtT5∈ℝ77×4096 the encoder of a T5-v1.1-XXL model. cctxtCLIP can then be zero-filled along the channel axis on 4096 dimensions to match the T5 representation, and along the sequence axis with cctxtT5 are chained together to form a final context representation, c ctxt ∈ R 154×4096 , to obtain. c ctxt is represented as prompt conditioning 220.

[0055] The second set of prompt coding models 218 can generate one or more encodings of the prompt 208. In the illustrated example, the second set of prompt coding models 218 includes a CLIP-G / 14 coding model, a CLIP-L / 14 coding model, and a T5-XXL coding model. In certain embodiments, more or fewer coding models are used. In certain embodiments, a different combination of coding models is used. The second set of prompt coding models 218 can include one or more coding models that were used during the training of the exemplary content creation system 400. In the illustrated embodiments, each of the coding models included in the second group of prompt coding models 218 generates an encoding of at least a part of the prompt 208.

[0056] The exemplary content creation system 400 illustrates that the prompt conditioning 220 can undergo linear operations in addition to the operations described above before being used by the reverse diffusion transformer 224. Linear operations were described above in relation to the linear 306a operations. The exemplary content creation system 400 illustrates that the combined encoding 212 can be input into an MLP 402a. The operations of an MLP were described in detail above. The output of MLP 402a can be passed to an encoding addition system 214 to be added to an encoded time step 206, output by the time-step encoding model 204, to produce a time condition 216 (described in more detail above).

[0057] The coded time step 206 can be generated by the time-step coding model 204. The time-step coding model 204 can generate a coded time step 206 using a received time step 202. The time-step coding model 204 can perform a sine coding 406 and use an MLP 402b. MLP 402b can perform operations similar to the MLPs described above. The sine coding 406 can involve using a sine function to determine an output coding based on the time step 202. The output of the sine coding 406 can be represented in a vector space.

[0058] A noisy latent space 222 can also be received and / or generated by an exemplary content creation system 400 to serve as input for the reverse diffusion transformer 224. The noisy latent space 222 can be a latent pixel representation. The rasterized latent space 222 can be generated by adding (e.g., using the encoding addition system 214) a positional encoding 412 to an initial rasterized latent space 408 or a latent space derived using the initial rasterized latent space 408 (e.g., after patch 410 operations and / or linear 404b operations have been performed on the initial rasterized latent space 408). Patch-410 operations can create patches (e.g., 2 × 2 patches) of the noisy latent pixel representation x ∈ ℝ h×w×cflatten to a patch coding sequence of length 0.5 * h * 0.5 * w. The noisy latent space 222 can have a common dimensionality as prompt conditioning 220.

[0059] The reverse diffusion transformer 224 can use the time conditioning 216, the prompt conditioning 220, and the noisy latent space 222 as inputs to generate the content 230. The reverse diffusion transformer 224 can contain any number of transformer blocks 414 (e.g., MM-DiT blocks 414a, 414b to 414d). In one embodiment, the size of the reverse diffusion transformer 224 is parameterized with respect to the model depth d (e.g., the number of attention blocks) by setting the hidden size to 64 * d (extended to 4 * 64 * d channels in the MLP blocks) and the number of attention heads equal to d. The reverse diffusion transformer 224 and the transformer blocks 414 can use the time conditioning 216 and the prompt conditioning 220 as inputs for the modulation mechanism(s). The transformer blocks 414 are described further herein (e.g., with respect to the Fig. 3 and Fig. 4B).

[0060] After a number of blocks contained in the reverse diffusion transformer 224 have received inputs and generated corresponding outputs, the last transformer block 414d can cause the output of a conditioned latent space 322. The conditioned latent space 322 can be output by the reverse diffusion transformer 224. The conditioned latent space 322 can be subjected to modulation operations 416 (e.g., the modulation operations described above), leaner operations 404c, and / or unpatching operations 418. The unpatching operations 418 serve to upsample the conditioned latent space 322 into a dimensionality that the decoding model 228 (e.g., as described above) can use to generate the content 230.

[0061] Fig. Figure 4B shows an exemplary transformer block 414 (e.g., 414a, 414b to 414d) contained in a reverse diffusion transformer 224 according to embodiments of the present disclosure. The exemplary transformer block 414 can receive the time conditioning 216, the prompt conditioning 220, and / or the noisy latent space 222 as input and use the inputs to generate a conditioned prompt 320 and / or a conditioned latent space 322. The conditioned prompt 320 and the conditioned latent space 322 have been described in detail above and can be used in subsequent transformer blocks (e.g., in the exemplary transformer block 414d) and / or as output of the transformer model (e.g., in the reverse diffusion transformer 224). The conditioned latent space 322 can be decoded by a decoding code model to generate the content 230.

[0062] The exemplary transformer block 414 can perform Swish activation function (SiLu) operations 420a and linear 404d operations on the time conditioner 216 to produce a first time conditioner result 424a. The first time conditioner result 424a can be used to condition the vector space being worked with in order to produce the conditioned prompt 320. Similarly, separate operations (SiLu 420b and linear 404e) can be performed on the time conditioner 216 to produce a second time conditioner result 424b. The first temporal conditioning outcome 424a and the second temporal conditioning outcome 424b may not be the same because the prompt modality (e.g., text, prompt conditioning 220) is weighted differently compared to the content modality (e.g., image, noisy latent space 222) (e.g., based on training).As shown, the values ​​of the first time conditioning result 424a and the second time conditioning result 424b can be used during different steps of the operations performed by the exemplary transformer block 414.

[0063] The prompt conditioning 220 can be subjected to other operations, such as layer norm operations 302a, modulation operations 304a, linear operations 306a, and / or root mean squared (RMS) normalization operations 422, before being used as input for a chaining system 308. The chaining system can use the input values ​​to generate values ​​that are transmitted to the joint self-monitoring system 310. The joint self-monitoring system 310 can generate an output by using inputs connected to the prompt conditioning 220 and the noisy latent space 222. The normalization operations 422 can prevent instability of the attentional logit and enable efficient training.If the model uses the normalization operations 422 in addition to the operations of the layer norm 302a, the normalization operations 422 can help to stabilize large models.

[0064] The output of the common self-attention system 310 can be used to perform linear 312a operations, multiple coding operations using a coding multiplication system 426, coding addition operations using the coding addition system 214, layer norm 314a operations, modulation 316a operations, MLP 318a operations, additional multiple coding operations using a coding multiplication system 426, and / or coding addition operations using the coding addition system 214 at the conditioned prompt 320. The coding multiplication system can use values ​​generated using time conditioning 216 and an intermediate value from the sequence of operations to produce an output.The coding addition system 214 can use the conditioning of the prompt 220 and an intermediate value from the sequence of operations to generate the output.

[0065] Similar operations can be performed on the noisy latent space 222 as were performed on the prompt conditioning 220, but with the noisy latent space 222 instead of the prompt conditioning 220. These similar operations can generate the conditioned latent space 322.

[0066] Fig. Figure 5 shows an example of a system 500 for training a diffusion transformer according to embodiments of the present disclosure. The diffusion transformer model can include a forward diffusion transformer 520 (e.g., a system or a model) and a reverse diffusion transformer 224, which can be configured by a transformer tuning system 528 during a training process. The forward diffusion transformer 520 can include a system configured to perform a process of adding noise to a latent training content 518 (e.g., a latent basic truth) to generate a noisy latent training content 522. On the other hand, the reverse diffusion transformer 224 can be configured to generate latent output contents 526 using a coded prompt conditioning 506, a time conditioning 536 and / or noisy latent training contents 522.The reverse diffusion transformer can be the reverse diffusion transformer 224 described above.

[0067] In some embodiments, the architecture of the forward diffusion transformer 520 can be compared to the reverse diffusion transformer model in the above. Fig. The architecture described in section 2 should be similar, but configured to generate a noisy latent training content 522 using latent training content 518.

[0068] Before the reverse diffusion transformer 224 can be used during inference time, it can first be trained to reverse the noise introduced by the forward diffusion transformer 520. The forward diffusion transformer 520 can introduce noise into the latent training content 518 to generate noisy latent training content 522, so that the reverse diffusion transformer 224 can learn how to reverse the noise introduced by the forward diffusion transformer 520. The forward diffusion transformer 520 uses the latent training content 518, which was generated by encoding the training content 514 with the encoder model 516. The latent training content 518 can be considered the basic truth for training purposes.The forward diffusion transformer 520 generates an increasingly noisy latent training content 522 and passes this noisy latent training content 522 to the reverse diffusion transformer 224 to reverse the added noise and attempt to obtain the latent training content 518 from the noisy latent training content 522. The forward diffusion transformer 520 can add noise to the latent training content 518 by sampling from a Gaussian distribution to obtain a vector of the same size as the latent training content 518 and then interpolating between the latent training content 518 and the noise data based on coefficients derived from a sampled time step value.

[0069] The training of the reverse diffusion transformer 224 and / or the forward diffusion transformer 520 can be performed using a training dataset containing pairs of content (e.g., training content 514) and prompts (e.g., training prompt 502). For example, the training data in the training dataset might contain a training prompt 502 stating, "Photo of a bear in a suit and top hat in a river in the middle of a forest, holding a sign that says 'I can't stand it'," and a corresponding training content 514 containing a picture of a bear in a suit and top hat in a river in the middle of a forest, holding a sign that says "I can't stand it." The training data may have been generated using a cluster-based deduplication method to remove perceptual and semantic duplicates from the original training data.

[0070] The training content 514 and the training prompt 502 can each be used during the training process of the Reverse Diffusion Transformer 224. The training prompt 502 contained in the training data can also be used during the training of a prompt coding model contained in the first set of prompt coding models 508 and / or the second set of prompt coding models 504 (e.g., before a first set of prompt coding models 508 and / or a second set of prompt coding models 504 is used to perform inference-time calculations to generate the coded prompt conditioning 506, the combined coding 510, the combined coding 212, and / or the prompt conditioning 220).

[0071] The training content 514 can be used with an encoder model 516 to generate latent training content 518. The forward diffusion transformer 520 can generate the noisy latent training content 522 using the latent training content 518. The noisy latent training content 522 can be received by the encoder model 516. The encoder model 516 may have been trained to generate a latent output representation of the input content it receives. The latent training content 518 may be a high-dimensional representation of the training content 514. The encoder model 516 may have been trained for content in general or for a specific type of content (e.g., images, videos, images of a particular style, images with a particular mood, images with particular colors, etc.).

[0072] At least a portion of the training prompt 502 contained in the training data can be input into the first set of prompt coding models 508 and / or the second set of prompt coding models 504. The first set of prompt coding models 508 can be the first set of prompt coding models 210 described above or any other set of prompt coding models. The second set of prompt coding models 504 can be the second set of prompt coding models 218 described above or any other set of prompt coding models. One or more of the prompt coding models contained in a set of prompt coding models can have been trained to generate an encoding of a prompt (e.g., training prompt 502) or a portion of the prompt. The 502 training prompt can contain text, audio, video and / or an image, etc.The coded prompt conditioning 506, generated by the second set of prompt coding models 504, can be used by the reverse diffusion transformer 224 to generate the latent output content 526. The combined coding 510, generated by the first set of prompt coding models 508, can be used by the coding addition system 214 to generate a time conditioning 536, which can then be used by the reverse diffusion transformer 224 to generate the latent output content 526. The coding addition system 214 can be the coding addition system 214 described above and add the combined coding 510 with a coded time step 534.

[0073] The encoded time step can be determined by a time step encoding model 204 (e.g., the one in Fig. 2 and Fig. The time-step encoding model 204 described in Section 4A is generated based on a time step 512. The time steps can be sampled non-uniformly. In certain embodiments, the time steps are sampled at a higher frequency between an initial time step and an end time step. For example, the sampling of time steps can follow an approximate normal distribution or a logit normal distribution. The time steps and the time-step encoding model have already been described in more detail here. By using a higher sampling frequency in the middle steps compared to the early and late steps, where the inverse diffusion process is very heavy or very light, the generated output can be more accurate (e.g., the input prompt can be reproduced more accurately).

[0074] The noisy latent training content 522, generated by the forward diffusion transformer 520, can also be used as input for the reverse diffusion transformer 224 to train the reverse diffusion transformer 224 to generate the latent output content 526 based on the noisy latent training content 522. The reverse diffusion transformer 224 can use the coded prompt conditioning 506, the time conditioning 536, and the noisy latent training content 522 to learn how a training prompt 502 corresponds to the latent training content 518, which was used to generate the noisy latent training content 522. The Reverse Diffusion Transformer 224 can use the coded prompt conditioning 506 and the time conditioning 536 to perform conditioning (e.g., cross-attention conditioning, self-attention conditioning).The learning / training can be performed over many iterations. During the iterations, the parameter values ​​of the reverse diffusion transformer 224 and / or the forward diffusion transformer 520 (e.g., if a forward diffusion transformer model system is used) can be adjusted using transformer weight matching signals 530 from the transformer matching system 528.

[0075] The transformer tuning system 528 can compare the latent output content 526, generated by the reverse diffusion transformer 224, and the latent training content 518 (e.g., a basic truth) to determine transformer weight tuning signals 530, which are transmitted to the forward diffusion transformer 520 (e.g., in embodiments where the forward diffusion transformer 520 is a model) and / or the reverse diffusion transformer 224. The transformer matching system 528 can use a loss function to compare latent training content 518 and latent output content 526. The transformer matching system 528 can transmit the transformer weight tuning signals 530 with the aim of minimizing the loss function. The loss is used to generate gradients to train the transformer(s) during backpropagation.

[0076] In some embodiments, the reverse diffusion transformer 224 uses images of a first size and performs fine-tuning using images of a second size that is larger than the first size and / or images with mixed aspect ratios.

[0077] In certain embodiments, the training prompt 502 and the training content 514 are derived from a predefined set of training data. In certain embodiments, the predefined set of training data may be limited in the size and / or attributes of the training prompt 502 corresponding to the training content 514. At least a portion of the training data and the training content 514 may be used to synthetically generate training prompts 502. A vision-language model (e.g., CogVLM) may be used to generate synthetic training prompts 502. The training of the reverse diffusion transformer 224 may include training that utilizes at least a portion of the generated / synthetic training prompts 502 and the corresponding training content 514.In certain embodiments, the training data includes prompts from an original dataset and synthetically generated prompts using the technique described above. The training data may contain a split between original and synthetic prompts (e.g., a 50 / 50 split). Synthetically generated prompts can be used to enhance the image output produced by the Reverse Diffusion Transformer 224. The synthetic labels can enable models that use text to generate images to be trained on more data and / or on data with different characteristics. For example, human-generated prompts may use a different language and / or focus on different descriptions than a prompt generated by a visual language model.

[0078] Fig. Figure 6 shows an example of a system 600 for training an encoding model (e.g., encoding model 516) and / or a decoding model (e.g., decoding model 228), according to embodiments of the present disclosure. The system 600 may comprise an encoding model 516, the decoding model 228, and an automatic encoder tuning system 608. The decoding model 228 and / or the decoding model 516 can be trained by using the autoencoder tuning system 608 to compare the training content 602 of the basic truth with the output content 606 generated by the decoding model 228 and to adjust the weights of the decoding model 516 and / or the decoding model 228 based on the comparison. Through training iterations, the decoding model 228 can learn to generate accurate output content 606 using latent training content 604. The latent training content 604 can have the same dimensions as a conditioned latent space (e.g.,conditioned latent space 226, latent output content 526), ​​which is generated by models with reverse diffusion transformers, which are related to the above. Fig. 1-5 are described, so that the decoding model 228 can generate contents 230 using the conditioned latent space 226 created by the reverse diffusion transformer 224.

[0079] The coding model 516 may have been trained by the system 600 and / or be trained to generate an embedding of the basic truth training content 602. The training content 602 may be an image, a video, or other content. The training content 602 may contain the training content (e.g., in a pixel space) used for the training described above. The coding model 516 may process the training content 602 through a series of convolution blocks, each of which performs downsampling. All convolutions may be parameterized in a weight-normalized form. The coding model 516 may map an input to a low-dimensional space. Increasing the number of latent channels may improve reconstruction performance. The reconstruction quality of the coding model 516 may represent an upper limit to the achievable image quality after latent diffusion training.For example, having 16 latent channels might offer an improvement over an 8-channel coding model. The number of channels can also be balanced, keeping in mind that fewer channels allow the model to use fewer resources. As the number of latent channels increases, the benefit may decrease.

[0080] The latent training content 604, generated by the encoding model 516, can be transferred to the decoding model 228 to be used as input for the decoding model 228. The decoding model 228 can be trained by system 600 to output generated output content 606 (e.g., in a pixel space) based on latent training content 604 (e.g., an encoding of the basic truth training content 602). The output content 606 can be an image, a video, or other content. The architecture of the decoding model 228 can be similar to the architecture of the encoding model 516, but it uses upsampling blocks. The architecture of the decoding model 214 is described above in relation to... Fig. 3 described.

[0081] The autoencoder adaptation system 608 can use output content 606 and training content 602 (a basic truth) to determine weighting adjustment signals 610 to be sent to the encoder model 516 and / or the decoder model 228. The autoencoder adaptation system 608 can compare the output content 606 and the training content 602 using a loss function. In some embodiments, a reconstruction loss function is used. Based on the comparison of the output content 606 and the training content 602 using the loss function, the autoencoder adaptation system 608 can transmit the weighting adjustment signals 610 to the encoder model 516 and / or the decoder model 228 with the aim of reducing the loss function. In some embodiments, the 608 autoencoder matching system uses an adverse loss term that employs a convolutional discriminator model. The discriminator model may include hyperparameters.{ {Inventor, can an adverse loss term be used for training the autoencoder? What are some loss functions that can be used?}}.

[0082] Fig. Figure 7 shows an example of a process 700 for the use of a content creation system (e.g. the content creation system 108 described above) according to embodiments of the present disclosure.

[0083] In step 702, a prompt is received from the content creation system. The prompt can be received by a computer system (e.g., computer system 104). The prompt can describe one or more desired properties of content to be generated by the content creation system. For example, the desired properties can include style, color, theme, mood, texture, contrast, depth, motion, saturation, focus, perspective, narrative, and / or other characteristics. The prompt can contain sample content (e.g., images, videos), audio, text, and video. The content of a prompt can serve as inspiration for content creation. The prompt can contain content that the system can add to and / or modify to generate content.

[0084] In step 704, an initial prompt encoding (e.g., combined encoding 212) can be generated. The initial prompt encoding can be generated by inputting the prompt into an initial set of one or more encoding models. The encoding models in the initial set can each generate an encoding of at least one part of the prompt and output the encoding of that part, which can then be combined with other encodings from the initial set to generate the initial prompt encoding. The initial set of encoding models can contain one or more text encoders. The initial set of encoding models can be used for one or more parts of the prompt. For example, the initial set of encoding models could include a CLIP-G / 14 model and a CLIP-L / 14 model.

[0085] In step 706, a second prompt encoding (e.g., prompt conditioning 220) can be generated. The second prompt encoding can be generated by inputting the prompt into a second set of one or more encoding models. The encoding models in the second set can each generate an encoding of at least one part of the prompt and output the encoding of that part, which can then be combined with other encodings from the second set to generate the second prompt encoding. The second set of encoding models can contain one or more text encoders. The second set of encoding models can be used for one or more parts of the prompt. For example, the second set of encoding models can include a CLIP-G / 14 model, a CLIP-L / 14 model, and / or a T5 XXL model.

[0086] The first set of coding models and the second set of coding models can contain one or more of the same coding models (e.g., with the same parameters and the same architecture). The first set of coding models and the second set of coding models can also contain one or more coding models that are not included in the second set or the first set, respectively. In some embodiments, the first set of coding models and / or the second set of coding models is trained using more coding models than the respective set uses at inference time.

[0087] In step 708, a first conditioned latent space and a second conditioned latent space are generated. The first conditioned latent space can be referred to as a conditioned prompt (e.g., conditioned prompt 320). The first conditioned latent space can be generated by a first transformer block of a diffusion transformer model. The first transformer block can generate the first conditioned latent space based on the first prompt encoding, the second prompt encoding, and a noisy latent space. The noisy latent space can share a common dimensionality with the second prompt encoding.

[0088] The first prompt encoding, or any encoding derived from it, can be combined (e.g., added) with a time-step encoding to generate a time-conditioning signal (e.g., Time Conditioning 216). The time-conditioning signal can contain a time-step encoding. The first conditioned latent space can be generated by the first transformer block using the time-conditioning signal (e.g., Time Conditioning 216).

[0089] The second conditioned latent space can be referred to as a conditioned latent space (e.g., conditioned latent space 322). The second conditioned latent space can be generated by the first transformer block of the diffusion transformer model. The first transformer block can generate the second conditioned latent space based on the first prompt encoding, the second prompt encoding, and the noisy latent space. The second conditioned latent space can also be generated by the first transformer block using the time-conditioning signal.

[0090] The first transformer block can perform reverse diffusion transformations on the first prompt encoding and the noisy latent space. The reverse diffusion operations can be informed by the time-conditioning signal. The first transformer block can use a shared self-attention system (e.g., shared self-attention system 310) to work together with intermediate values ​​generated from the first prompt encoding and the noisy latent space, respectively. The output of the shared self-attention system can be further processed in two independent sets of operations to generate the first conditioned latent space and the second conditioned latent space in their respective domains. The independently performed sets of operations can use independent weighting (e.g., different weights).

[0091] The output of the first transformer block can be used as input for a subsequent transformer block. Within the diffusion transformer model, any number of transformer blocks can be chained together. The transformer blocks can contain a linear layer trained with machine learning low-rank matrices (LoRA). The last block in the chain is described in connection with step 710.

[0092] In step 710, a third conditioned latent space is generated. This third conditioned latent space can be generated by a second transformer block included in the diffusion transformer model. The second transformer block can have a similar architecture to the first transformer block. The third conditioned latent space can be generated based on the output of a previous transformer block. For example, the third conditioned latent space can be generated based on the first and second conditioned latent spaces. The third conditioned latent space can also be generated based on the time-conditioning signal produced by encoding a current time step.Since the second transformer block can have a similar architecture to the first, it can be configured to generate a conditioned prompt signal and a conditioned latent space signal. The second transformer can output the conditioned latent space signal, which is then passed to a decoding model (e.g., Decoding Model 228). The conditioned latent space signal can contain an encoded representation of the content described in the prompt.

[0093] In step 712, the decoding model can be used to decode the encoded conditioned latent spatial representation of the content described in the prompt. The content may contain one or more features described in the prompt.

[0094] In some embodiments, after the content has been generated, the content generation system can receive a subsequent request that causes a second piece of content to be generated, which differs from the first and is based on the first request and / or the first content. For example, the second request can require that one or more features be added, further emphasized, removed, or modified.

[0095] Fig. Figure 8 shows an example of a process 800 for training an encoder model (e.g., encoder model 516, time-step encoder model 204) and / or a decoder model (e.g., decoder model 228), according to embodiments of the present disclosure. The encoder and / or decoder model can be trained using the system described in relation to Fig. 6 is described above.

[0096] In step S802, an encoder model (e.g., encoder model 516) can receive training content, which is used to generate a corresponding latent training space representation of the training content. The training content can be contained within the training data. The training content and training data have already been described in detail above. In one example, the first training content comprises images and / or videos and corresponds to a training prompt.

[0097] In step S804, the decoder model can be used to generate the output content using the latent spatial representation of the training content.

[0098] In step S806, the output content generated by the decoder model (e.g., using an autoencoder matching system 608) can be compared with the training content input into the encoder model to determine the similarity of the content. The comparison can be performed using a loss function (e.g., a reconstruction loss function, KL divergence loss, adverse loss, or perceptual loss). Additionally or alternatively, the comparison can be performed using a discriminator model, which were described in more detail above. A discriminator model can be used when a watermark is added to generated images.

[0099] KL divergence loss can be used with variational autoencoders (VAEs). KL divergence loss can be used to measure the difference between a learned latent distribution and a previous distribution (often a standard Gaussian distribution). This stimulates the latent space to follow a particular distribution, enabling better generative abilities.

[0100] Adversarial loss can be used with adversarial autoencoders (AAEs), which introduce a discriminator network alongside the autoencoder. The adversarial loss is used to match the latent spatial distribution with a desired prior distribution. This can be similar to the loss used in GANs (Generative Adversarial Networks).

[0101] Instead of simple pixel-by-pixel reconstruction, perceptual loss compares high-level features extracted from pre-trained networks (such as VGG) between the original and the reconstructed images. This can be useful for tasks like super-resolution, where perceptual quality is more important than exact pixel-level accuracy.

[0102] In step S808, the weights of the encoding model and / or the decoding model can be adjusted based on the comparison performed in step S806. The weight adjustment can be performed to minimize the loss function or otherwise to make the output content more similar to the training content.

[0103] Steps S802–S808 can be repeated over a number of training epochs to train or fine-tune the decoder model and / or the encoder model. Once the decoding model has been trained, it can be used during inference time to generate content based on a prompt (e.g., as part of the Content Generation System 108). The decoder parameters can be frozen after training and before use during inference time. In some embodiments, the encoder model, after being trained using the procedure described above, is used during the training process of the diffusion transformer model. The encoder parameters can be frozen after training and before use during inference (e.g., during the training of the diffusion transformer model).

[0104] Fig. Figure 9 shows an example of a process 900 for training a diffusion transformer model according to embodiments of the present disclosure. The diffusion transformer model can include a forward diffusion transformer (e.g., forward diffusion transformer 520) and a reverse diffusion transformer (e.g., reverse diffusion transformer 224). The reverse diffusion transformer can be used during inference time by a content generation system (e.g., the content generation system 108 described above).

[0105] In step S902, training content can be received from a coding model (e.g., coding model 516), and the coding model can generate latent training content. The training content can be contained within the training data and correspond to a training prompt. The training content, the data, and the prompt have already been described in detail above.

[0106] In step S904, the latent training content of the training content can be entered into the forward diffusion transformer (e.g., a system or a model) to cause the forward diffusion transformer to generate noisy latent training content (e.g., noisy latent training content 522).

[0107] In step S906, a training prompt (e.g., training prompt 502) corresponding to the first training content (e.g., describing attributes of the training content) can be received for use in step S908 and / or step S910. The training prompt can be included in the training data set. The training data set can be the same training data set used by the coding model in step S902. In some embodiments, the training prompt describes the desired properties of the tone to be generated. For example, the desired properties can include style, color, theme, mood, texture, contrast, depth, motion, saturation, focus, perspective, narrative, and / or other attributes. The training prompt can contain audio, image, video, and / or speech data used to generate latent output content (e.g., latent output content 526).The training prompt may contain content that is to be added and / or modified by the content creation system.

[0108] In step S908, a coded prompt conditioner (e.g., coded prompt conditioner 506) can be created. The coded prompt conditioner can be output by at least one prompt coding model. The coded prompt conditioner can contain a combination of outputs from the at least one prompt coding model. The at least one prompt coding model can be contained in a set of one or more prompt coding models (e.g., a second set of prompt coding models 504). The coded prompt conditioner can be passed to a reverse diffusion transformer model (e.g., reverse diffusion transformer 224) for use in training the reverse diffusion transformer.

[0109] In step S910, a combined prompt / prompt encoding (e.g., combined encoding 510) can be generated. The combined prompt encoding can be generated using one or more encoding models from a set of one or more encoding models (e.g., an initial set of prompt encoding models 508). The outputs of encoding models contained in the set of one or more encoding models can be combined to generate the combined prompt encoding. The combined prompt encoding can be passed to an encoding addition system to be added to an encoded time step. The encoded time step can be generated by encoding a time step using a time step encoding model (e.g., time step encoding model 204). The combined prompt encoding is then added to the time step encoding to generate a conditioning vector.The conditioning vector can be used by a reverse diffusion transformer (e.g., reverse diffusion transformer 224) to generate latent output contents.

[0110] In step S912, the reverse diffusion transformer can use the noisy latent training content, the conditioning vector, and the encoded prompt conditioning to generate the latent output content. The reverse diffusion model can contain one or more transformer blocks operating in two separate domains to perform reverse diffusion in each domain with separate weighting, while simultaneously performing joint self-attention operations (e.g., QKV attention, multi-head attention) between the two domains. During the joint self-attention operation, prompt embedding can be used to help the model focus on specific parts of an image embedding and to capture further contextual relationships between the prompt embedding and the prompt embedding.

[0111] In step S914, the noisy latent training content generated in step S904, the first coded prompt conditioning generated in step S908, and / or the conditioning vector generated in step S912 can be input into a reverse diffusion transformer. The reverse diffusion transformer can use the inputs to generate latent output content (e.g., latent output content 526). The reverse diffusion transformer can use the inputs to condition its performance as described above (e.g., with respect to...). Fig. 2-5). The reverse diffusion transformer can be configured to output a latent content into a conditioned latent space. The generated latent output content can be representative of the training content.

[0112] In step S916, the latent output content generated in step S914 can be compared with the latent training content generated in step S902. This comparison can be performed by a transformer fitting system (e.g., Transformer Fitting System 528). A loss function can be used to compare the latent training content and the latent output content.

[0113] In step S918, a transformer weight setting signal can be transferred to the forward diffusion transformer and / or the reverse diffusion transformer (e.g., via the transformer setting system) based on the comparison performed in step S916. Step 918 can be referred to as reverse propagation. Step S918 may be performed in an effort to minimize a loss function used during training process 900, thereby increasing the accuracy of the reverse diffusion transformer.

[0114] Steps S902–S918 can be repeated over a number of training epochs to train or fine-tune the reverse diffusion transformer and / or the forward diffusion transformer. As training progresses, the noisy latent training content generated by the forward diffusion transformer can be made increasingly noisy (e.g., so that the noise more closely resembles a Gaussian distribution). With further training, the reverse diffusion transformer can become increasingly accurate in generating a latent output content (similar to the latent training content) based on the noisy latent training content, time conditioning, and the initial coded prompt conditioning it receives as input.Once the reverse diffusion transformer has been trained, the trained reverse diffusion transformer can be used during inference time to generate content based on a prompt (e.g., as part of the content generation system 108).

[0115] As a result of the training process, the content creation system can receive a reverse diffusion transformer that uses separate and respective weights for two modalities and a bidirectional information flow between tokens for each modality to generate content with improved prompt understanding and / or typography.

[0116] Fig. Figure 10 is a simplified block diagram illustrating an example architecture of a System 1000 used for training and / or using the models and systems described herein, according to some embodiments.

[0117] System 1000 comprises a computer system 104, a network 1008, and a server 1004. The computer system 104 can correspond to any of the user devices and / or computer systems described herein. The server 1004 can correspond to one or more server computers (e.g., a server cluster) of a cloud computing platform, as described herein.

[0118] Network 1008 can encompass any suitable communication path or channel, such as a line or cable, fiber optic cable, telephone line, cellular connection, radio frequency connection, WAN or LAN network, the internet, or any other suitable medium. Network 1008 can include any one or a combination of many different types of networks, such as wired networks, the internet, wireless networks, cellular networks, and other private and / or public networks. The network can utilize infrared, ultra-wideband (UWB), Bluetooth (BT), Bluetooth Low Energy (BTLE), Wi-Fi, and / or radio communication technologies.

[0119] As regards the individual elements, the computer system 104 can be any suitable computing device (e.g., a mobile phone, a tablet, a PC, smart glasses, a smartwatch, etc.). The computer system 104 has at least one memory 1010, one or more processing units (or processor(s)) 1014, a storage unit 1016, a communication interface 1018, and one input / output device 1020.

[0120] The 1014 processor(s) can be implemented in hardware, computer-executable instructions, firmware, or a combination thereof, as required. Computer-executable instructions or firmware implementations of the 1014 processor(s) can contain computer-executable or machine-executable instructions written in a suitable programming language to perform the various described functions.

[0121] Memory 1010 can store program instructions that can be loaded and executed by the processor(s) 1014, as well as data generated during the execution of these programs. Depending on the configuration and type of computer system 104, memory 1010 can be volatile (e.g., random access memory (RAM)) and / or non-volatile (e.g., read-only memory (ROM), flash memory, etc.). In some implementations, memory 1010 may include several different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), or ROM. The computer system 104 may also include additional memory 1016, such as either removable or non-removable memory, including but not limited to magnetic storage, optical disks, and / or tape storage.The disk drives and their associated computer-readable media can provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for computer equipment. In some embodiments, the memory can be used to store audio, video, image, and / or text files.

[0122] The computer system 104 may also include the communication interface 1018, which enables the user device 1002 to communicate with the server, user terminals, and / or other devices in the network(s) 1008. The computer system 104 may also include I / O device(s) 1020, for example, to enable connection to a keyboard, mouse, pen, speech input device, touch input device, display, speakers, printer, and / or other components that the computer system 104 may contain.

[0123] As regards the contents of memory 1010 in detail, memory 1010 may contain an operating system and one or more application programs or services for implementing the features disclosed herein, including a content generation system 108 or a system for training one or more of the models used in the content generation system 108.

[0124] It should be understood that one or more functions of the content generation system 108 can be executed by the computer system 104 and / or the server 904.

[0125] In some embodiments, as described above, the remote server 1004 can correspond to a cloud computing platform. The remote server 1004 can perform one or more functions, including, for example: receiving a prompt; generating a first prompt encoding; generating a second prompt encoding; generating a first conditional latent space and a second conditional latent space based on the prompt encoding(s) and a first latent space; generating a third conditional latent space based on the first conditional latent space and the second conditional latent space; and / or generating content based on the third conditional latent space. The remote server 1004 can transmit the content to the computing system 104. The remote server 1004 can include a module for generating credentials, I / O devices, and / or communication interfaces, etc.contain.

[0126] Regarding the contents of memory 1030 in detail, memory 1030 may contain an operating system 1032 and one or more application programs or services for implementing the functions disclosed herein, including a communication module 1034, an encryption module 1036, the content generation system 108 and / or a profile management module 1040.

[0127] The communication module 1034 can contain code that instructs the processor 1046 to receive requests, create embeds, train models, transfer content, and / or otherwise communicate with other system components. For example, the communication module 1034 can receive requests and transfer content to the computer system 104.

[0128] The encryption module 1036 can contain code that instructs the processor 1046 to encrypt and / or decrypt messages. For example, the encryption module 1036 can receive encrypted data (e.g., requests) from the computer system 104. The encryption module 1036 can contain any suitable encryption algorithms for encrypting data. Suitable data encryption algorithms can be Data Encryption Standard (DES), Triple DES, Advanced Encryption Standard (AES), etc. It can also store encryption keys (e.g., encryption and / or decryption keys) that can be used with such encryption algorithms (e.g., in the memory unit 1048). The encryption module 1036 can use symmetric or asymmetric encryption techniques to encrypt and / or verify data.For example, computer system 104 may contain a code and / or key similar to encryption module 1036, which is suitable for encrypting / decrypting data communication with computer system 104 (and / or server 1004).

[0129] The profile management module 1040 can contain code that instructs the processor 946 to maintain and store profiles of users and / or user devices. For example, the profile management module 940 can receive information about users and / or devices that are authorized to use the content generation system 108 and / or train the content generation system 108. The profile management module 1040 can track users and / or devices associated with prompts and / or generated content, so that when the users and / or devices reuse the server 1004, the prompts and / or generated content can be transferred back to them (e.g., displayed as a content generation history). The profile management module 1040 can also contain information about which users and / or user devices have which permissions, etc.

[0130] The in the Fig. 7-9 shown (and / or in relation to the Fig. The processing described in sections 1-6 and all other figures can be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of the respective systems using hardware or combinations thereof. The software can be stored on a non-transient storage medium (e.g., a storage device). The methods presented here are to be understood as illustrative and not limiting. Although in the Fig. Figures 7-9 and others depict the various processing steps in a specific order; this should not be interpreted as a restriction. In certain alternative embodiments, the processing may be carried out in a different order, or some steps may be performed in parallel. It should be noted that in alternative embodiments, the steps shown in the Fig. The processing shown in figures 7-9 and other figures may involve a greater or lesser number of steps than those shown in the respective figures.

[0131] The various embodiments can further be implemented in a variety of operating environments, which in some cases may include one or more user computers, computing devices, or processing equipment that can be used to run any number of applications. User or client devices may include any number of general-purpose personal computers, such as desktop or laptop computers running a standard operating system, as well as cellular, wireless, and handheld devices running mobile software and capable of supporting a range of network and messaging protocols. Such a system may also include a number of workstations running a variety of commercially available operating systems and other well-known applications for purposes such as development and database management. These devices may also include other electronic equipment, such as...Dummy terminals, thin clients, gaming systems and other devices that can communicate over a network.

[0132] Most embodiments use at least one network known to those skilled in the art to support communication using a variety of commercially available protocols, such as Transmission Control Protocol / Internet Protocol (“TCP / IP”), Open Systems Interconnection (“OSI”), File Transfer Protocol (“FTP”), Universal Plug and Play (“UPnP”), Network File System (“NFS”), Common Internet File System (“CIFS”), and AppleTalk. The network could be, for example, a local area network, a wide area network, a virtual private network, the Internet, an intranet, an extranet, a public telephone network, an infrared network, a wireless network, or any combination thereof.

[0133] In implementations that use a web server, the web server can run a variety of server or mid-tier applications, including Hypertext Transfer Protocol (HTTP) servers, FTP servers, Common Gateway Interface (CGI) servers, data servers, Java servers, and servers for business applications. The server(s) can also execute programs or scripts in response to requests from user devices, for example, by running one or more web applications, which can be implemented as one or more scripts or programs written in any programming language, such as Java. ® The servers are written in C, C#, or C++, or any scripting language such as Perl, Python, or TCL, as well as combinations thereof. The servers may also include database servers, especially those from Oracle. ® , Microsoft ® , Sybase ® , and IBM ® commercially available.

[0134] The environment can include a variety of data storage devices and other storage and storage media, as described above. These can be located in different places, for example, on a storage medium that is located locally on (and / or resident in) one or more computers, or remotely from one or all computers via the network. In a particular embodiment, the information can be stored in a Storage Area Network (“SAN”) known to those skilled in the art. Likewise, all files required for the execution of the functions assigned to the computers, servers, or other network devices can be stored locally and / or decentrally, as needed. If a system includes computerized devices, each of these devices can contain hardware elements that can be electrically coupled via a bus, the elements being, for example, at least one central processing unit (“CPU”), at least one input device (e.g., a keyboard, mouse, or keyboard), and so on.A system may include a mouse, keyboard, controller, touchscreen, or keypad and at least one output device (e.g., a display, printer, or speaker). Such a system may also include one or more storage devices such as hard disk drives, optical storage devices, and solid-state storage devices such as random-access memory (RAM) or read-only memory (ROM), as well as removable media, memory cards, flash cards, etc.

[0135] Such devices may also include a computer-readable storage device, a communications device (e.g., a modem, a network card (wireless or wired), an infrared communications device, etc.), and working memory as described above. The computer-readable storage device may be connected to or configured to receive computer-readable storage media. These may be remote, local, fixed, and / or removable storage devices, as well as storage media that temporarily and / or permanently contain, store, transmit, and retrieve computer-readable information. The system and the various devices typically also include a set of software applications, modules, services, or other elements residing in at least one working memory, including an operating system and application programs, such as a client application or a web browser.It should be acknowledged that alternative embodiments may exhibit numerous variations of those described above. For example, custom hardware could be used, and / or certain elements could be implemented in hardware, software (including portable software such as applets), or both. Furthermore, a connection to other computer devices, such as network input / output devices, may be used.

[0136] Storage media: Computer-readable media containing programs / code or parts of programs / code may include all suitable media known or used in the art, including storage media and communication media, such as volatile and non-volatile, removable and non-removable media, which are stored and / or transmitted by any method or technology, such as...Computer-readable instructions, data structures, program modules, or other data, including RAM, ROM, electrically erasable programmable read-only memory (“EEPROM”), flash memory or other storage technologies, compact disc read-only memory (“CD-ROM”), digital versatile disk (DVD), or other optical storage media, magnetic cartridges, magnetic tapes, magnetic disk storage media, or other magnetic storage devices, or any other medium that can be used to store the desired information and that a system device can access. Based on the disclosure and the teachings contained herein, a person with ordinary technical knowledge will estimate other ways and / or methods for implementing the various embodiments.

[0137] The description and drawings are therefore to be understood in an illustrative rather than a restrictive sense. However, it will be evident that various modifications and changes can be made to them without deviating from the broader spirit and scope of the revelation as set forth in the claims.

[0138] Other variations are within the spirit of this disclosure. While the disclosed techniques are suitable for various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described in detail above. However, it should be understood that the intention is not to limit the disclosure to the specific form or forms disclosed, but rather, on the contrary, to cover all modifications, alternative constructions, and equivalents that fall within the spirit and scope of the disclosure, as defined in the attached claims.

[0139] The use of the terms "a," "a," and "the," and similar designations in connection with the description of the disclosed embodiments (particularly in connection with the following claims) is to be interpreted as encompassing both the singular and the plural, unless otherwise specified herein or clearly contradicted by the context. The terms "comprising," "with," "including," and "containing" are to be understood as open terms (i.e., in the sense of "including but not limited to"), unless otherwise specified. The term "connected" is to be understood as being wholly or partially contained in, attached to, or connected with something, even if something is in between.The specification of value ranges serves merely as a shorthand for each individual value falling within the range, unless otherwise stated herein, and each individual value is included in the description as if it were listed here individually. All procedures described herein may be carried out in any suitable order, unless otherwise stated herein or the context clearly advises against it. The use of examples or illustrative phrases (e.g., "such as") is solely for the purpose of better illustrating embodiments of the disclosure and does not constitute a limitation of the scope of the disclosure unless otherwise stated. No wording in the description should be interpreted as construing an unclaimed element as essential to the practice of the disclosure.

[0140] Disjunctive expressions such as "at least one of X, Y, or Z" are, unless expressly stated otherwise, to be understood in context as meaning that an object, concept, etc., can be either X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Therefore, such a disjunctive formulation should not and must not generally mean that in certain embodiments at least one of X, at least one of Y, or at least one of Z must be present.

[0141] Preferred embodiments of this disclosure are described herein, including the best way known to the inventors to carry out the disclosure. Variations of these preferred embodiments may be obvious to the person skilled in the art upon reading the foregoing description. The inventors expect that skilled craftsmen will use such variations appropriately, and the inventors intend to practice the disclosure differently than expressly described herein. Accordingly, this disclosure includes all modifications and equivalents of the subject matter specified in the appended claims, to the extent permitted by applicable law. Furthermore, any combination of the elements described above, in all possible variations thereof, is covered by the disclosure unless otherwise stated herein or clearly contradicted by the context.

[0142] All references, including publications, patent applications and patents cited herein, are hereby incorporated by reference to the same extent as if each reference were individually and expressly stated as incorporated by reference and set forth herein in its entirety. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] EP 25159213.5

[0001] US 63 / 567,127

[0001] US 63 / 633,020

[0001]

Claims

[1] A system that includes the following: one or more storage media that store instructions; and one or more processors configured to execute the instructions to cause the system to: to receive a request that describes a desired property of an image; to generate a prompt coding based on the prompt using a set of coding models; to generate a first prompt embedding and a first image embedding based on the prompt encoding and a noise input using a first transformer block of a diffusion transformer model; using a second transformer block of the diffusion transformer model, to create a second image embedding based on the first image embedding and the first prompt embedding; and to generate the image based on the second image embedding. [2] System according to claim 1, wherein the set of coding models comprises: a first subset of coding models; and a second subset of coding models that differs from the first subset of coding models. [3] System according to claim 2, wherein the first subset of coding models comprises a different number of coding models than the second subset of coding models. [4] System according to claim 1, wherein the set of coding models includes at least one of the following elements: a first text encoder trained together with an image encoder, or a second text encoder trained as a text-to-text encoder. [5] System according to claim 1, wherein the diffusion transformer model is trained using prompt embeddings generated by a second set of coding models that is different from the set of coding models. [6] System according to claim 1, wherein the generation of the first prompt embedding is further based on a first weighting contained in a first set of weightings assigned to a first area of ​​the first prompt embedding, and the generation of the first image embedding is further based on a second weighting contained in a second set of weightings assigned to a second area of ​​the first image embedding. [7] System according to claim 6, further comprising instructions which, when executed, cause the system to: to generate noisy latent input content using input content; to generate latent output content using the noisy latent input content and a third prompt encoding that corresponds to the noisy latent input content; and to adjust at least one of the first weightings or the second weightings based on the comparison of the noisy latent input content and the latent output content. [8] System according to claim 1, wherein generating the first image embedding or the first prompt embedding comprises applying an attention operation to an intermediate value generated on the basis of the noisy input and / or the prompt encoding. [9] System according to claim 1, wherein the diffusion transformer model is trained by at least: Pre-training of the diffusion transformer model using images of a first size; and Fine-tuning of the diffusion transformer model using at least: images with a second size that is larger than the first size, or images with mixed aspect ratios. [10] System according to claim 1, wherein the first prompt embedding and the first image embedding have a common dimensionality. [11] System according to claim 1, wherein the first transformer block and the second transformer block each contain a linear layer trained using machine learning low-rank matrices (LoRA). [12] System according to claim 1, wherein the diffusion transformer model is trained using training data containing a set of images and a set of prompts, wherein the set of prompts includes a synthetic prompt generated using a corresponding image from the set of images. [13] System according to claim 1, wherein the diffusion transformer model comprises a rectilinear flow model and is trained using a logit normal distribution. [14] System according to claim 1, further comprising instructions which, when executed, cause the system to: to generate a first vector space using the prompt encoding and a time step encoding; to generate a second vector space using the prompt encoding; and to generate the first prompt embedding using the first vector space, the second vector space, and the noise input; to generate the noise input using positional coding and noise pixel coding; and to generate the first image embedding using the first vector space, the second vector space, and the noise input. [15] System according to claim 14, wherein the time step is weighted more heavily if the time step is an intermediate step. [16] System according to claim 1, further comprising commands which, when executed, cause the system to: to create a text conditioning that contains encodings from at least two encoding models, using more than one encoding model included in the set of encoding models. [17] System according to claim 1, further comprising instructions which, when executed, cause the system to: to generate a first normalized intermediate value of the first request embedding; to generate a second normalized intermediate value of the first image embedding; to connect the first normalized intermediate value and the second normalized intermediate value; and to perform a self-awareness operation on the associated values. [18] System according to claim 1, wherein the prompt coding is a first prompt coding and the system further comprises instructions which, when executed, cause the system to: Generating a second prompt encoding using the prompt as input for a subset of encoding models contained in the set of encoding models; Generating a third prompt encoding using the prompt as input for a second subset of encoding models contained in the set of encoding models; and Generating the first request encoding by combining part of the second request encoding and the third request encoding. [19] One or more non-transitory computer-readable storage media that store instructions which, when executed by one or more processors of a system, cause the system to perform operations which include the following Receiving a prompt describing a desired property of an image; Generate, using a set of coding models, an input prompt encoding based on the input prompt; Generate, using a first transformer block of a diffusion transformer model, a first prompt embedding, and a first image embedding based on the prompt encoding and a noise input; Generate, using a second transformer block of the diffusion transformer model, a second image embedding based on the first image embedding and the first prompt embedding; and Generating the image based on the second image embedding. [20] Non-transitory computer-readable storage medium according to claim 19, wherein the diffusion transformer model is trained by at least: Pre-training of the diffusion transformer model using images of a first size; and Fine-tuning of the diffusion transformer model using at least: images with a second size that is larger than the first size, or images with mixed aspect ratios. [21] Non-transitory computer-readable storage medium according to claim 19, wherein the first prompt embedding and the first image embedding have a common dimensionality. [22] Non-transitory computer-readable storage medium according to claim 19, wherein the first transformer block and the second transformer block each contain a linear layer that is trained using low-rank learning matrices (LoRA). [23] Non-transitory computer-readable storage medium according to claim 19, wherein the diffusion transformer model is trained using training data containing a set of images and a set of prompts, wherein the set of prompts includes a synthetic prompt generated using a corresponding image from the set of images. [24] Non-transitory computer-readable storage medium according to claim 19, wherein the diffusion transformer model includes a rectilinear flow model and is trained using a logit normal distribution. [25] Non-transitory computer-readable storage medium according to claim 19, wherein the set of encoding models comprises: a first subset of coding models; and a second subset of coding models that differs from the first subset of coding models. [26] Non-transitory computer-readable storage medium according to claim 25, wherein the first subset of coding models contains a different number of coding models than the second subset of coding models. [27] Non-transitory computer-readable storage medium according to claim 19, wherein the set of coding models includes at least one of the following sets: a first text encoder trained together with an image encoder, or a second text encoder trained as a text-to-text encoder. [28] Non-transitory computer-readable storage medium according to claim 19, wherein the diffusion transformer model is trained using prompt embeddings generated by a second set of coding models that is different from the set of coding models. [29] Non-transitory computer-readable storage medium according to claim 19, wherein the generation of the first request embedding is further based on a first weight contained in a first set of weights associated with a first domain of the first request embedding, and the generation of the first image embedding is further based on a second weight contained in a second set of weights associated with a second domain of the first image embedding. [30] Non-transitory computer-readable storage medium according to claim 29, further comprising instructions which, when executed, cause the system to perform operations which further comprise: Generating noisy latent input content using input content; Generating latent output content using the noisy latent input content and a third prompt encoding that corresponds to the noisy latent input content; and Set at least one of the first weightings or the second weightings based on the comparison of the noisy latent input content and the latent output content. [31] Non-transitory computer-readable storage medium according to claim 19, wherein generating the first image embedding or the first prompt embedding comprises applying an attention operation to an intermediate value generated on the basis of the noisy input and / or the prompt encoding. [32] The non-transferable computer-readable storage medium according to prow 19, which further comprises instructions which, when executed, cause the system to carry out rations 15, which further comprise: Generating a first vector space using prompt encoding and time-step encoding; Generating a second vector space using the prompt encoding; and Generating the first prompt embedding using the first vector space, the second vector space, and the noise input; and wherein generating the first image embedding further comprises: Generating the noise input using positional coding and noise pixel coding; and Generating the first image embedding using the first vector space, the second vector space, and the noise input. [33] Non-transitory computer-readable storage medium according to claim 32, wherein the time step is weighted more heavily if the time step is an intermediate step. [34] The non-transitory, computer-readable storage medium according to claim 19, further comprising instructions which, when executed, cause the system to perform operations which further comprise: Generate, using more than one coding model included in the set of coding models, a text conditioning that contains encodings from at least two coding models. [35] Non-transitory computer-readable storage medium according to claim 19, further comprising instructions which, when executed, cause the system to perform operations which further comprise: Generating a first normalized intermediate value of the first request embedding; Generating a second normalized intermediate value of the first image embedding; Combining the first normalized intermediate value and the second normalized intermediate value; and Performing a self-awareness operation on the associated values. [36] Non-transitory computer-readable storage medium according to claim 19, further comprising instructions which, when executed, cause the system to perform operations which further include: Generating a second prompt encoding using the prompt as input for a subset of encoding models contained in the set of encoding models; Generating a third prompt encoding using the prompt as input to a second subset of encoding models contained in the set of encoding models; and Generating the first request encoding by combining part of the second request encoding and the third request encoding.

Images