One-step diffusion distillation via depth equilibrium model

Abstract

And performing one-step diffusion distillation through a depth equilibrium model. Generating a balanced converter is disclosed. The disclosed embodiments provide simple and efficient techniques that can use only noise / image pairs to distill a multi-step diffusion process into a single-step generative model.

Description

Technical Field

[0001] The present disclosure relates to distillation of diffusion models. In the disclosed embodiments, a deep balanced model may be utilized in a distillation architecture. The disclosed embodiments may enable distillation of a diffusion model directly from initial noise to a resulting image. Background Art

[0002] Diffusion models have shown excellent performance across a wide range of generative tasks, including high-quality image generation, audio and video synthesis. Knowledge distillation refers to the process of transferring knowledge from a large model or set of models to a single smaller (and faster) model that can be practically deployed under real-world constraints. Knowledge distillation can be viewed as a form of model compression. Summary of the invention

[0003] The main disadvantage of the diffusion model is that its slow generation process limits the practical applicability of the diffusion model in real-time or resource-constrained scenarios. Existing distillation methods for diffusion models aim to condense the multi-step sampling process into a more efficient few-step or single-step process. However, the disadvantages of these methods are that they require multiple passes of training to distill the lengthy sampling process, and they require a lot of memory and computing resources due to maintaining duplicate copies of the model. The disclosed embodiments can distill the multi-step diffusion process into a single-step generation model using only noise / image pairs.

[0004] In some embodiments, the disclosed method includes: converting noise into a noise embedding vector; tokenizing the noise embedding vector via an injection transformer; inputting the tokenized noise into a balun; solving a fixed point via the balun; and decoding the fixed point to generate an image sample. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 An example embodiment for generating image samples according to the present disclosure is illustrated.

[0006] Figure 2 An example embodiment for generating image samples according to the present disclosure is illustrated.

[0007] Figure 3 An example embodiment of a general computer system according to the present disclosure is illustrated. DETAILED DESCRIPTION

[0008] Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The figures are not necessarily drawn to scale; some features may be exaggerated or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as restrictive, but only as a representative basis for teaching those skilled in the art to adopt the present invention in various ways. As will be understood by those of ordinary skill in the art, the various features illustrated and described with reference to any one of the figures may be combined with the features illustrated in one or more other figures to produce embodiments that are not explicitly illustrated or described. The combination of the illustrated features provides representative embodiments of typical applications. However, for specific applications or embodiments, it may be desirable to carry out various combinations and modifications of features consistent with the teachings of the present disclosure.

[0009] Diffusion models have demonstrated remarkable performance on a wide range of generative tasks, such as high-quality image generation and manipulation, audio synthesis, video, 3D shapes, text, and molecule generation. These models are trained with denoising objectives derived from score matching, variational inference, or optimal transport, enabling them to generate clean data samples by progressively denoising the initial Gaussian noise during the inference process. Unlike adversarial training, the denoising objective leads to a more stable training process, which in turn allows diffusion models to scale effectively. Although the results are promising, a major drawback of diffusion models is their slow generation process, which typically necessitates hundreds to thousands of model evaluations. This computational complexity limits the applicability of diffusion models in real-time or resource-constrained scenarios.

[0010] To speed up the slow generation process of diffusion models, researchers have proposed distillation methods that aim to distill the multi-step sampling process into a more efficient few-step or single-step process. However, these techniques often come with their own set of challenges. The distillation objective must be carefully designed to successfully transfer knowledge from a larger model to a smaller model. Furthermore, distilling a long sampling process into a few-step process usually requires multiple training passes. Most of the popular online distillation techniques require maintaining a duplicate copy of the model, resulting in increased memory and computational requirements. As a result, there is a clear need for simpler and more efficient methods that address the computational requirements of distilled diffusion models without sacrificing generation capabilities.

[0011] One proposed knowledge distillation technique distills a multi-step denoising diffusion implicit model (“DDIM”) sampler into a single-step sampler by training a student model on one million synthetic image samples. Other techniques, such as step-wise distillation, aim to distill a T-step teacher DDIM into a new T / 2-step student DDIM, repeating this process until a one-step model is achieved. Transitive closure time-distillation generalizes step-wise distillation to simultaneously distill N>2 steps to reduce the total number of training stages. The consistency model achieves online distillation in a single pass by leveraging carefully designed teachers and distillation loss objectives. The Diffusion Fourier Neural Operator (“DFNO”) maps an initial Gaussian distribution to a solution trajectory of an inverse diffusion process by inserting a temporal Fourier integral operator in a pre-trained U-Net backbone. Another method proposes to distill a classifier-free guided diffusion model into a few-step generative models by first distilling the combined conditional and unconditional models, and then step-wise distilling the resulting model for faster generation.

[0012] While distillation can be an effective way to accelerate the sampling rate of existing diffusion models, there is an alternative industry that reduces the length of the sampling chain by considering alternative formulations of the diffusion model, correcting bias and truncation errors during the denoising process, and reducing the length of the sampling chain through training-free fast samplers at inference time. Several works, such as DDIM, improved DDPM, FastDPM, SGM-CLD, and EDM, modify or optimize the forward diffusion process so that the denoising process can be made more efficient. For example, DDIM is based on a non-Markov definition of the diffusion process with the same training purpose, but its inverse process is sampled faster. DPM-Solver and GENIE are high-order ordinary differential equation ("ODE") solvers that generate samples in a few steps. There are also methods to combine diffusion models with other families of generative models for faster sampling.

[0013] The purpose of the present disclosure is to simplify the distillation process of diffusion models while maintaining the perceptual quality of images generated by the original model. The disclosed embodiments may show enhancements over previous methods. The disclosed embodiments may include simple and effective techniques that distill a multi-step diffusion process into a single-step generative model using only noise / image pairs. The disclosed embodiments may include an architecture that includes a novel deep balanced ("DEQ") model, which may be referred to herein as a generative balanced transformer ("GET"). In some disclosed embodiments, GET can be interpreted as an infinite depth network using weight-bound transformer layers that solve fixed points in a forward pass. The disclosed embodiments allow these layers to be adaptively applied in the forward pass, thereby achieving a balance between inference speed and sample quality. In addition, the disclosed embodiments incorporate a nearly parameter-free class adjustment mechanism into the architecture, thereby extending its utility to class-conditional image generation.

[0014] The methods disclosed herein for distilling noise / image pairs generated via a diffusion model can be applied to both Visual Transformer ("ViT") and GET-based architectures. However, in the disclosed embodiments, the GET-based architecture achieves significantly improved results with a smaller architecture. In the context of both conditional and unconditional image generation, the GET-based architecture can produce perceptual image quality comparable to or better than other complex distillation techniques (such as stepwise distillation). In some embodiments, GET exhibits significantly better parameter and data efficiency compared to architectures such as ViT, as GET matches Frechet Inception Distance ("FID") scores that are 5 times larger than ViT, highlighting the transformative potential of GET in enhancing the efficiency of generative models.

[0015] The disclosed embodiments can be used to generate synthetic image datasets to train machine learning systems that can be used for any computer vision task, including: 1) computer-controlled machines, robots, vehicles, household appliances, power tools, manufacturing machines, personal assistants, and access control systems; and 2) systems for transmitting information, such as surveillance systems or medical (imaging) systems. The disclosed embodiments can also be used to enhance existing training datasets by generating additional images or by modifying images in an existing dataset.

[0016] The slow generation process of diffusion models requires hundreds of model evaluations, which limits their usefulness in real-time or resource-constrained scenarios. With the goal of accelerating the generation process, distillation methods have emerged that aim to distill the multi-step sampling process into a more efficient few-step or single-step process. These methods are often complex because they require multiple training stages, the resulting distilled models may not maintain the perceptual quality of images generated by the original model, and in many established methods, it is necessary to maintain a duplicate copy of the model, which in turn leads to increased memory.

[0017] The Deep Equilibrium ("DEQ") model (e.g., as described in the Deep Equilibrium model by Bai et al.) computes the internal representation by solving for fixed points in its forward pass. Specifically, consider a deep feed-forward model with L layers:

[0018]

[0019] in, is input injection, is the hidden state of the i-th layer, and is the feature transformation of the i-th layer, parameterized by θ. If the above model is weight-bound, that is, Then, under the limit of infinite depth, the output z[i] of such a network approaches a fixed point z ★ :

[0020]

[0021] DEQ solves for this fixed point z* directly in the forward pass using a black-box root-finding algorithm (such as Broyden's method) or Anderson acceleration. Since we cannot rely on explicit backpropagation through exact operations in the forward pass, DEQ uses implicit differentiation to perform analytical differentiation through fixed points. Let g θ (z ＊ ; x) = f θ (z ＊ ; x)-z ＊ , then the Jacobian matrix of z* with respect to the model weight θ is given by:

[0022]

[0023] When dealing with high-dimensional feature maps, computing the inverse of the Jacobian matrix can quickly become intractable. The inverse-Jacobian matrix term can be replaced with the identity matrix (i.e., no Jacobian matrix or an approximate inverse-Jacobian matrix) without affecting the final performance.

[0024] The diffusion model or the score-based model progressively perturbs the image with increasing amounts of Gaussian noise, and then reverses this process to generate the image by sequential denoising. Specifically, consider the iid sample dataset p data , then the diffusion process For t∈[0,T], The stochastic differential equation ("SDE") is given by:

[0025] dx=f(x,t)dt+g(t)dw (4)

[0026] Where w is a standard Wiener process, f(·, t): R d →R d is the drift coefficient, g(·):R→R is the diffusion coefficient, and x(0)~p data and x(T)~N(0,I). All diffusion processes have a corresponding deterministic process, called a probability flow ODE ("PF-ODE"), whose trajectory shares the same marginal probability density as the SDE. This ODE can be written as:

[0027]

[0028] where σ(t) is the noise schedule of the diffusion process, and represents the score function. It has been shown that the optimal choice of σ(t) in equation (5) is σ(t) = t. Therefore, the PF-ODE can be simplified to

[0029]

[0030] Among them, D θ (·, t) is a denoiser function parameterised by a neural network, which minimises the data The expected L2 denoising error of the samples extracted from . Samples can be efficiently generated from this ODE by numerical methods such as Euler's method, Runge-Kutta method, and Heun's second-order solver.

[0031] The disclosed embodiments include a one-step distillation diffusion model, referred to herein as GET. GET may include a DEQ visual transformer that can distill the diffusion model into generative models that can quickly sample images using only a single model evaluation. The GET model can distill a Gaussian noise set e and an optional (given by Figure 1 The dashed box in the figure indicates that the class label y is mapped to the image

[0032] Reference Figure 1 , an embodiment of the GET-based architecture 100 may include an embedding network 110 for embedding noise 112, an embedding network 120 for embedding class labels 122, an injection transformer (“InjectionT”) 130, and a balancing transformer (“EquilibriumT”) 140. The embedding network 110 may embed the noise e 104 into an embedding vector h 114, and the embedding network 120 may embed the class label y 122 into an embedding vector c, as shown according to equation (7).

[0033] h, c=Emb(e), Emb(y);ify≠0 (7)

[0034] The noise embedding network 110 can convert the input noise e∈R HxWxC Convert to a sequence of two-dimensional (“2D”) patches Where C is the number of channels, P is the patch size, H and W indicate the height and width of the original image, and is the number of resulting patches. By adding the standard sinusoidal positional encoding, a noise embedding vector h is generated. Since the entire generation process is directly distilled, the temporal embedding t common in standard diffusion models is not required.

[0035] According to equation (8), the injection transformer 130 can transform the tokenized noise (e.g., the embedding vector h) into an intermediate representation n. Figure 1, the output is input to an optional linear component 132. The linear component 132 can perform a linear projection of the injection transformer 130 output for computing an intermediate representation n. Alternatively, the linear component can be omitted or can be included in the injection transformer 132, which will then output an intermediate representation n. This intermediate representation n can be the input of the balun 140. According to equation (9), the balun 140 can be a balancing layer. The balun 140 can solve for the fixed point z* by introducing a noise injection n and an optional class embedding vector c.

[0036] n=InjectionT(h,c) (8)

[0037] z*=EquilibriumT(z*,n,c) (9)

[0038] The injection transformer 130 and the balancing transformer 140 may be composed of a series of transformer blocks (ie, GET blocks) having similar block designs, such as Figure 2 As shown in FIG. 1 . Both the injection transformer 130 and the balancing transformer 140 may consist of a series of GET blocks. The injection transformer 130 may be called only once to generate the noise injection n. The balancing transformer 140 defines the implicit layer z * =f θ (z * , n, c) θ , this function can be called multiple times to create a weight-bound computational graph until convergence.

[0039] The GET block can use similar block designs for noise injection (i.e. InjectionT) and balancing layers (i.e. EquilibriumT), which may differ only at the injection interface. Specifically, the GET block can be built on the standard Pre-LN Transformer block as shown below:

[0040] z=z+Attention(LN(z),u)

[0041] z=z+FFN(LN(z),u).

[0042] Here, z∈R NxD represents a potential token, u∈R Nx3D is the input injection, LN, FFN, and Attention represent layer normalization, a 2-layer feed-forward network with hidden dimension size D×E, and an attention layer with injection interface, respectively.

[0043] For the GET block used in the injection transformer 130, u is equal to the class embedding token c∈R for conditional image generation. 1x3DThat is, for the conditional model, u = c, otherwise u = 0. For the GET block in the balun, u is the noise injection n∈R NxD and class embedding token c∈R 1x3D That is, for the conditional model, u = n + c, otherwise u = n.

[0044] The standard Transformer attention layer can be modified to NxD , key k∈R nxD The sum v∈R NxD Previously incorporated into the additive injection interface,

[0045] q,k,v,=zW i +u

[0046] z=MHA(z,k,v)

[0047] z=zW。

[0048] Among them, W i ∈R Dx3D and W o ∈R DxD The injection interface enables interaction between latent tokens and input injection in a multi-head dot-product attention ("MHA") operation.

[0049]

[0050] Among them, W q , W k ∈R DxD is a slice from Wi, and u q , W k ∈R DxD is a slice from u. Compared with the standard MHA operation, this scheme does not add more computational cost, but it achieves similar effects as criss-cross attention and provides good stability during training.

[0051] The output z* of balun 140 may be decoded and rearranged to generate image samples according to equation (10) 160. The decoder 150 may include a layer normalization component and a linear layer to generate small blocks The resulting small pieces can be rearranged back to the resolution of the input noise e to produce image samples

[0052]

[0053] Figure 2An example embodiment 200 for generating image samples according to the present disclosure is illustrated. Figure 2 In the example, the embedding network 202 embeds the noise e into an embedding vector h (not shown), and the embedding vector h is input into the injection transformer. For example, the injection transformer can be Figure 1 The injection transformer 130 illustrated in FIG. The injection transformer may include one or more GET blocks 214. The embedding vector h may be input into the GET block after the layer normalization component 206 of the GET block. For example, the embedding network 202 may be Figure 1 The same embedded network 110 shown in FIG. Figure 2 , the optional (indicated by the dashed box) embedding network 204 embeds the class label y into the embedding vector c, and the embedding vector c is input into the injection transformer after the normalization layer 206.

[0054] In this embodiment 200, the injection transformer may include one or more GET blocks 214. Each GET block 214 includes a layer normalization component 206, 210. Layer normalization is a technique used in deep learning to normalize the activations (outputs) of neural network layers. It normalizes each training example independently across its features, thereby reducing the dependency on batch size. The layer normalization component 206 receives as input an embedding vector h (not shown) from the embedding network 202.

[0055] Each GET block 214 includes a multi-head attention ("MHA") component 208. Multi-head attention is a component used in Transformer-based models. The attention mechanism allows the model to focus on different parts of the input sequence when making predictions or generating outputs. Multi-head attention extends this idea by using multiple sets of attention weights (called "attention heads") to capture different types of relationships and information in the input sequence. The MHA component 208 can receive as input the output from the layer normalization component 206. The MHA component 208 can optionally receive as input an embedding vector c.

[0056] Each GET block 214 includes a second layer normalization component 210. The layer normalization component 210 may receive the output of the MHA component 208 as input.

[0057] Each GET block 214 includes a multi-layer perception ("MLP") component 212. The MLP can be composed of linear layers combined with non-linear activation functions. The linear layers perform transformations involving weights and biases, while the non-linear activation functions introduce non-linearity into the network. The MLP component 212 can receive as input the output from the second layer normalization component 210.

[0058] exist Figure 2In the example, the output n of the injection transformer is input to the balun. For example, the balun can be Figure 1 The balun 140 illustrated in FIG. 1 may include one or more GET blocks 220 . Figure 2 The balun in may optionally receive an embedding vector c as an additional input n. Figure 2 As illustrated in FIG. 1 , one or more GET blocks 220 have a structure similar to the GET block 214 in the injection transformer. That is, the GET block 220 has a first layer normalization component 222 similar to the layer normalization component 206, an MHA component 224 similar to the MHA component 208, a second layer normalization component 226 similar to the layer normalization component 210, and an MLP component 228 similar to the MLP component 212.

[0059] Figure 2 The fixed point z* can be decoded and rearranged to generate image samples. For example, the fixed point z* can be provided to a decoder such as Figure 1 The decoder 150 generates image samples

[0060] Figure 3 An example embodiment of a general computer system 300 according to the present disclosure is illustrated. The computer system 300 may include a set of instructions that may be executed to cause the computer system 300 to perform any one or more of the methods or computer-based functions described herein. For example, the computer system 300 may include executable instructions for executing Figure 1 and Figure 2 The computer system 300 may be connected to other computer systems or peripheral devices via a network. Additionally, the computer system 300 may include or be included within other computing devices.

[0061] like Figure 3As shown in , the computer system 300 may include one or more processors 302. For example, the one or more processors 302 may include one or more central processing units (CPUs), one or more graphics processing units (GPUs), or both. The computer system 300 may include a main memory 304 and a static memory 306, which communicate with each other via a bus 308. As shown, the computer system 300 may further include a video display unit 310, such as a liquid crystal display (LCD), a projection television display, a flat panel display, a plasma display, or a solid-state display. Additionally, the computer system 300 may include an input device 312, such as a remote control device having a wireless keypad, a keyboard, a microphone coupled to a speech recognition engine, a camera (such as a video camera or a still camera), or a cursor control device 314 (such as a mouse device). The computer system 300 may also include a disk drive unit 316, a signal generating device 318 (such as a speaker), and a network interface device 320. The network interface 320 may enable the computer system 300 to communicate with other systems via a network 328. For example, network interface 320 may enable machine learning system 120 to communicate with a database server (not shown) or a controller in a manufacturing system (not shown).

[0062] In some embodiments, Figure 3 , the disk drive unit 316 may include one or more computer-readable media 322 in which one or more instruction sets 324, such as software, may be embedded. For example, the instructions 324 may embody one or more methods or functions, such as those disclosed herein. In certain embodiments, the instructions 324 reside completely or at least partially within the main memory 304, the static memory 306, and / or the processor 302 during execution by the computer system 300. The main memory 304 and the processor 302 may also include computer-readable media.

[0063] In some embodiments, dedicated hardware implementations, such as application specific integrated circuits, programmable logic arrays, and other hardware devices, may be constructed to implement one or more methods or functions described herein. Applications that may include the apparatus and systems of various embodiments may broadly include various electronic and computer systems. One or more embodiments described herein may implement functions using two or more specific interconnected hardware modules or devices having associated control and data signals that may be transmitted between and through the modules or as part of an application specific integrated circuit. Thus, the present system encompasses software, firmware, and hardware implementations or combinations thereof.

[0064] Although the computer-readable medium is shown as a single medium, the term "computer-readable medium" includes a single medium or multiple media, such as a centralized or distributed database and / or associated caches and servers storing one or more instruction sets. The term "computer-readable medium" shall also include any medium capable of storing or encoding an instruction set that is executed by a processor or causes a computer system to perform any one or more of the methods or functions disclosed herein.

[0065] In some embodiments, some or all of the computer readable media will be non-transitory media. In a specific non-limiting, exemplary embodiment, the computer readable medium may include a solid-state memory, such as a memory card or other package that accommodates one or more non-volatile read-only memories. Further, the computer readable medium may be a random access memory or other volatile rewritable memory. Additionally, the computer readable medium may include a magneto-optical or optical medium, such as a disk or tape or other storage device, for capturing a carrier signal, such as a signal transmitted on a transmission medium.

[0066] Disclosed herein is a Generative Balanced Transformer ("GET"), a deep balanced visual transformer well suited for single-step generative models. The unique architecture of GET allows for a trade-off between inference speed and quality of generated images. Also disclosed herein is a simplified diffusion distillation process, and it is disclosed that directly training noise / image pairs from diffusion models (with the GET architecture) is an effective strategy for distilling multi-step sampling chains into single-step generative models, both for class-conditional and unconditional cases. One or more disclosed embodiments show that implicit models for generative tasks can strictly outperform classical networks in terms of performance, model size, model computation, training memory, and speed.

[0067] Although exemplary embodiments are described above, these embodiments are not intended to describe all possible forms covered by the claims. The words used in the specification are descriptive words, not restrictive words, and it is understood that various changes can be made without departing from the spirit and scope of the present disclosure. As previously described, the features of each embodiment can be combined to form further embodiments of the present invention that may not be explicitly described or illustrated. Although each embodiment may have been described as providing advantages or being superior to other embodiments or prior art implementations in terms of one or more desired characteristics, it is recognized by those of ordinary skill in the art that one or more features or characteristics can be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, strength, durability, marketability, appearance, packaging, size, applicability, weight, manufacturability, ease of assembly, etc. As such, in terms of one or more characteristics, embodiments described as less desirable than other embodiments or prior art implementations are not outside the scope of the present disclosure and may be desirable for specific applications.

Claims

1. A method comprising: Convert the noise into a noise embedding vector; tokenizing the noise embedding vector via an injection transformer; inputting the tokenized noise into the balun; solving for fixed points via a balun; and Decode the fixed points to generate image samples. The method of claim 1 , wherein the injection transformer comprises a series of transformer blocks.

3. The method according to claim 2, wherein each converter block comprises: A first-layer normalization component that receives the noisy embedding vector; The second layer normalization component; A multi-head attention component between the first layer normalization component and the second layer normalization component; as well as Multi-layer perception components. The method of claim 1 , wherein the balun comprises a series of transformer blocks.

5. The method according to claim 4, wherein each converter block comprises: A first-layer normalization component that receives the tokenized noise; The second layer normalization component; A multi-head attention component between the first layer normalization component and the second layer normalization component; as well as Multi-layer perception components.

6. The method of claim 1, wherein the decoding is performed by a decoder comprising a layer normalization component and a linear layer.

7. The method according to claim 1, further comprising: Convert class labels into class embedding vectors; Prior to tokenization of the noisy embedding vector, the class embedding vector is fed into the injection transformer; as well as Prior to the solution of the fixed points, the class embedding vector is input into the balun.

8. A non-transitory memory comprising computer executable instructions that, when executed by a system, cause the system to perform operations comprising: Convert the noise into a noise embedding vector; tokenizing the noise embedding vector via an injection transformer; inputting the tokenized noise into the balun; solving for fixed points via a balun; and Decode the fixed points to generate image samples.

9. The memory of claim 8, wherein the injection transformer comprises a series of transformer blocks.

10. The memory of claim 9, wherein each converter block comprises: A first-layer normalization component that receives the noisy embedding vector; The second layer normalization component; A multi-head attention component between the first layer normalization component and the second layer normalization component; as well as Multi-layer perception components.

11. The memory of claim 8, wherein the balun comprises a series of transformer blocks.

12. The memory of claim 11, wherein each converter block comprises: A first-layer normalization component that receives the tokenized noise; The second layer normalization component; A multi-head attention component between the first layer normalization component and the second layer normalization component; as well as Multi-layer perception components.

13. The memory of claim 8, wherein the decoding is performed by a decoder comprising a layer normalization component and a linear layer.

14. The memory of claim 8, wherein the operations further comprise: Convert class labels into class embedding vectors; Prior to tokenization of the noisy embedding vector, the class embedding vector is fed into the injection transformer; as well as Prior to the solution of the fixed points, the class embedding vector is input into the balun.

15. A system comprising: one or more processors; as well as a non-transitory memory comprising computer executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising: Convert the noise into a noise embedding vector; tokenizing the noise embedding vector via an injection transformer; inputting the tokenized noise into the balun; solving for fixed points via a balun; and Decode the fixed points to generate image samples.

16. The system of claim 15, wherein the injection transformer comprises a series of transformer blocks.

17. The system of claim 16, wherein each converter block comprises: A first-layer normalization component that receives the noisy embedding vector; The second layer normalization component; A multi-head attention component between the first layer normalization component and the second layer normalization component; as well as Multi-layer perception components.

18. The system of claim 15, wherein the balun comprises a series of transformer blocks.

19. The system of claim 18, wherein each converter block comprises: A first-layer normalization component that receives the tokenized noise; The second layer normalization component; A multi-head attention component between the first layer normalization component and the second layer normalization component; as well as Multi-layer perception components.

20. The system of claim 15, wherein the decoding is performed by a decoder comprising a layer normalization component and a linear layer.

Images