Target generation method, system and device based on diffusion Transformer and medium
By optimizing the computation process of the diffusion Transformer through CFG-aware cascaded difference calculation and recalculation of the output reference value decoding technology, the problems of high computational complexity and large storage requirements are solved, and efficient image, video and speech generation is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SEMICON TECH INNOVATION CENT(BEIJING) CORP
- Filing Date
- 2026-01-09
- Publication Date
- 2026-04-21
AI Technical Summary
The high computational complexity of the Diffusion Transformer is mainly due to its iterative nature and the large amount of computation required by the Classifier-free Guidance method. Furthermore, the existing inter-step difference calculation requires a large amount of DRAM storage and memory access, resulting in high transmission latency and energy consumption, which limits its application scope.
By employing CFG-aware cascaded difference calculation technology and recomputation-based output reference value decoding technology, the input and output of the previous step are cached as reference values, reducing DRAM storage and memory access. The difference calculation is optimized by utilizing the similarity between steps and batches.
It significantly reduces DRAM storage and memory access, lowers transmission latency and energy consumption, improves computing efficiency and throughput, and enables fast and efficient image, video and voice generation.
Smart Images
Figure CN121902873A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology, and in particular relates to a target generation method, system, device and medium based on diffusion Transformer. Background Technology
[0002] In object generation tasks, thanks to its excellent generation quality and diversity, Diffusion Model (DM) has become one of the most competitive generative model paradigms and is widely used in image, video and speech generation tasks. Early DMs were based on the U-Net structure, but the latest works are based on the Transformer structure, namely DiffusionTransformer (DiT).
[0003] However, on the other hand, DiT's computational complexity is enormous, limiting its application scope. This is mainly due to two reasons: 1) Iterative characteristics: DM, including DiT, generates content through a reverse denoising process. Each time content is generated, DiT needs to be run in multiple steps, and each step requires running the entire DiT network completely; 2) CFG (Classifier-Free Guidance): DM, including DiT, usually uses CFG to control the generation of content. For each step of each content generation, two batches need to be run, with the generation target and empty labels as conditional inputs respectively.
[0004] To address this problem, inter-step Delta Computing has been proposed, such as... Figure 1 As shown, the computational cost is reduced by leveraging the similarity between steps. For all linear layers in all batches of the DiT network, the input and output of the previous step are cached as reference values. In the current step, the current input is subtracted from the cached input reference to obtain the difference. Then, only matrix multiplication is performed on the difference, and the result is added to the output reference value of the previous step to obtain the current output. Due to the similarity between steps, the differences are often very sparse, containing many zero values and low-order values. Performing mixed-precision sparse matrix multiplication only on the differences can significantly reduce the computational cost.
[0005] Inter-step interpolation calculations require storing the inputs and outputs of all linear layers from the previous step. This data is enormous and has a very long lifespan, spanning the entire model's runtime, making it impossible to store in on-chip SRAM. Therefore, existing inter-step interpolation calculations inevitably lead to additional off-chip DRAM accesses, which causes additional transmission latency and power consumption, limiting the speedup potential and energy efficiency. Summary of the Invention
[0006] The purpose of this invention is to provide a target generation method, system, device, and medium based on diffusion Transformer to solve the problems existing in the prior art.
[0007] To achieve the above objectives, this invention provides a target generation method based on diffusion Transformer, comprising:
[0008] Obtain task requirements, which include requirements for generating images, videos, and audio.
[0009] The task requirements are input into the target generation model, and the corresponding generation results are output. The target generation model is built based on the diffusion Transformer model and consists of multiple stacked Transformer Blocks. Each Transformer Block contains a multi-head attention layer and a multi-layer perceptron layer. In the target generation process, the target generation model uses randomly generated Gaussian noise as the original input and combines time steps and the Classifier-free Guidance method to perform the content generation task.
[0010] Optionally, in the target generation process, the target generation model uses randomly generated Gaussian noise as the original input and performs multi-step denoising. Each denoising step requires the complete execution of the target generation model to obtain the final generation result. The multi-step denoising process is numbered T, T-1, ..., 1 in sequence. The number of the current denoising step is t. When t=T, the target generation model uses the original Gaussian noise and generation conditions as input. When t is not T, the target generation model uses the output result of step t-1 and generation conditions as input.
[0011] Optionally, during the target generation process, the Classifier-free Guidance method can be used to generate the content to be generated.
[0012] Optionally, the generation of the content to be generated using the Classifier-free Guidance method includes the following specific steps:
[0013] For each generated object, each denoising process requires running two batches. One batch uses the category or text description of the generated object as the generation condition, and the other batch uses empty labels as the generation condition. The output of each step is obtained by linear extrapolation of the results of the two batches.
[0014] Let the two batches be b0 and b1. For any linear layer, let the corresponding weight be W. Let the input corresponding to b0 and b1 at time t be x.t [0], x t [1], the output is denoted as y t [0], y t [1] The calculation process of the current linear layer is as follows:
[0015] When t=T, for b0, input x t [0] Store in DRAM, then calculate the output as y t [0]=Wx t [0], y t [0] Store in DRAM; for b1, calculate the input difference dx=x between b1 and b0. t [1]-x t [0], then perform linear calculation dy=Wdx, and further combine the output of b0 to obtain the complete result y. t [1]=dy+y t [0];
[0016] When t is not equal to T, and Tt is odd: For b0, load the input x from the previous step from DRAM. t+1 [0] and output y t+1 [0], find the difference dx = x between the current input and the previous input. t [0]-x t+1 [0] Perform linear calculation dy=Wdx, and further combine it with the output of the previous step to obtain the complete result y. t [0]=dy+y t+1 [0]; For b1, input x t [1] Store in DRAM, calculate the input difference dx=x between b1 and b0. t [1]-x t [0], then perform linear calculation dy=Wdx, and further combine the output of b0 to obtain the complete result y. t [1]=dy+y t [0], then y t [1] Stored in DRAM;
[0017] When t is not equal to T, and Tt is even: For b1, load the input x from the previous step from DRAM. t+1 [1] and output y t+1 [1] Find the difference dx = x between the current input and the previous input. t [1]-x t+1 [1] Perform linear calculation dy=Wdx, and further combine it with the previous output of b0 to obtain the complete result y. t [1]=dy+y t +1[1]; For b0, input it into x t[0] Store in DRAM, calculate the input difference dx = xt[0] - x t [1] Then perform linear calculation dy=Wdx, and further combine the output of b1 to obtain the complete result y. t [0]=dy+y t [1], then output y t [0] Stored in DRAM.
[0018] Optionally, when there are two linear layers l1 and l2, and at time t+1 the inputs and outputs of l1 and l2 need to be stored in DRAM, and in subsequent steps t the inputs and outputs of l1 and l2 need to be loaded from DRAM, optimization is performed: at time t+1, the input of l2 is not stored in DRAM, and at time t, the input of l2 is not loaded from DRAM. Based on the output of l1, the input of l2 is recalculated through the activation function.
[0019] A target generation system based on diffusion Transformer, comprising:
[0020] The task requirement determination module is used to obtain task requirements, which include the generation requirements of images, videos and audio.
[0021] The target generation module is used to input the task requirements into the target generation model and output the corresponding generation results. The target generation model is built based on the diffusion Transformer model and consists of multiple stacked Transformer Blocks. Each Transformer Block contains a multi-head attention layer and a multi-layer perceptron layer. In the target generation process, the target generation model uses randomly generated Gaussian noise as the original input and combines time steps and the Classifier-free Guidance method to perform the content generation task.
[0022] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor running the computer program to cause the electronic device to perform the aforementioned diffusion-based Transformer-based target generation method.
[0023] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned target generation method based on diffusion Transformer.
[0024] The technical effects of this invention are as follows:
[0025] This invention utilizes CFG-aware cascaded interpolation calculation technology to solve the problem that existing inter-step interpolation calculation technologies require storing the input and output of all batches of linear layers; it also utilizes recomputation-based output reference value decoding technology to solve the problem that existing inter-step interpolation calculation technologies require storing the input and output of all layers of linear layers; thus, it can significantly reduce the required DRAM storage and memory access, thereby reducing transmission latency and transmission energy consumption, ultimately improving throughput and energy efficiency, and achieving fast and efficient content generation, which can be applied to image, video, and multimodal scenarios. Attached Figure Description
[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0027] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0028] Figure 1 This is a schematic diagram illustrating the calculation of inter-step differences;
[0029] Figure 2 This is a schematic diagram of the target generation model in an embodiment of the present invention;
[0030] Figure 3 This is a schematic diagram of the cascaded difference calculation of CFG sensing in an embodiment of the present invention;
[0031] Figure 4 This is a schematic diagram of output reference value decoding based on recalculation in an embodiment of the present invention;
[0032] Figure 5 This is a flowchart illustrating the implementation of an embodiment of the present invention. Detailed Implementation
[0033] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features, and embodiments of the present invention.
[0034] It should be understood that the terminology used in this invention is merely for describing particular embodiments and is not intended to limit the invention. Furthermore, with respect to numerical ranges in this invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Every smaller range between any stated value or intermediate value within a stated range, and any other stated value or intermediate value within said range, is also included in this invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.
[0035] Various modifications and variations can be made to the specific embodiments described in this specification without departing from the scope or spirit of the invention, as will be apparent to those skilled in the art. Other embodiments derived from this specification will also be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.
[0036] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.
[0037] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0038] like Figure 1 - Figure 5 As shown, this embodiment provides a target generation method based on diffusion Transformer, including: obtaining task requirements, which include image, video, and speech generation requirements; inputting the task requirements into a target generation model and outputting the corresponding generation results; wherein, the target generation model is built based on the diffusion Transformer model and consists of multiple stacked Transformer Blocks, each Transformer Block containing a multi-head attention layer and a multi-layer perceptron layer; during the target generation process, the target generation model uses randomly generated Gaussian noise as the original input and combines time steps and a Classifier-free Guidance method to perform the content generation task.
[0039] This embodiment can reduce the computational complexity of DiT without causing a large number of DRAM memory accesses and additional storage, thereby improving overall throughput and energy efficiency.
[0040] The specific technical solution of this embodiment includes:
[0041] 1) CFG-aware cascaded difference calculation technology combines inter-step difference calculation and inter-batch difference calculation, making full use of inter-step similarity and inter-batch similarity; compared with existing inter-step difference calculation technology, it is no longer necessary to store the reference values of all batches in DRAM.
[0042] 2) Output reference value decoding technology based on recalculation: For adjacent linear layers separated by activation functions, only the input of the previous layer is cached and the output of the next layer is not cached when calculating the inter-step difference. When the latter is needed, it is obtained by recalculating the former. Compared with the existing inter-step difference calculation technology, it is no longer necessary to store the input reference values of all layers in DRAM.
[0043] This embodiment utilizes CFG-aware cascaded interpolation calculation technology to solve the problem that existing inter-step interpolation calculation technologies require storing the input and output of all batches of linear layers; it also utilizes output reference value decoding technology based on recomputation to solve the problem that existing inter-step interpolation calculation technologies require storing the input and output of all layers of linear layers; thus, it greatly reduces the required DRAM storage and memory access, thereby reducing transmission latency and transmission energy consumption, ultimately improving throughput and energy efficiency, and achieving fast and efficient content generation, which can be applied to image, video and multimodal scenarios.
[0044] like Figure 2 The diagram shown is a schematic of the model structure in this embodiment. The DiT model consists of multiple stacked Transformer Blocks, each of which contains a multi-head attention layer and a multilayer perceptron (MLP) layer.
[0045] In the generation process, randomly generated Gaussian noise is used as the original input, and denoising is performed in multiple steps. Each step requires a complete run of the DiT model to obtain the final generated result. The noise is numbered sequentially as T, T-1, ..., 1; the current step number is t. When t=T, the DiT model uses the original Gaussian noise and generation conditions as input; when t is not T, the DiT model uses the result of step t-1 and the generation conditions as input.
[0046] The DiT model uses the Classifier-Free Guidance (CFG) method to generate content. For each generated object, each step requires running two batches: one batch uses the object's category or text description as the generation condition, and the other batch uses an empty label as the generation condition. The output of each step is obtained by linear extrapolation of the results from these two batches.
[0047] The Cascaded Difference Calculation Process of CFG-Aware: Currently, mainstream DM typically uses the Classifier-Free Guidance (CFG) method for content generation. Each step requires running two batches, one with the generated object and the other with an empty label. Experimental observations show that the intermediate feature values between the two batches involved in CFG also exhibit similarity. By performing difference calculations on them, the computational load can be greatly reduced. However, since batches can be interleaved, the lifetime of the inter-batch reference values in the on-chip SRAM is extremely short, eliminating the need to store them in DRAM.
[0048] This calculation process combines the inter-step similarity and inter-batch similarity present in the DiT generation process. On the one hand, it can effectively reduce computational complexity, and on the other hand, it saves DRAM storage and memory access compared to existing solutions, thus better balancing computation and memory access.
[0049] The cascaded difference calculation process of CFG sensing is as follows: Assume the inverse denoising process of DiT has a total of T steps, numbered T, T-1, ..., 1 in sequence; denote the current step number as t. For the two batches involved in CFG, assume they are numbered b0 and b1 respectively; for any linear layer, denote its weight as W, and denote the inputs corresponding to b0 and b1 at step t as x. t [0], x t [1], the output is denoted as y t [0], y t [1]. In the inverse denoising process of DiT, this linear layer is calculated according to the following procedure:
[0050] When t=T: For b0, input it into x t [0] Store in DRAM, then calculate its output as y t [0]=Wxt[0], y t [0] Store in DRAM; for b1, calculate the input difference dx=x between it and b0. t [1]-x t [0], then perform a linear calculation dy=Wdx, and then add the output of b0 to get the complete result y. t [1]=dy+y t [0];
[0051] When t is not equal to T, and Tt is odd: For b0, load the input x from the previous step from DRAM. t+1 [0] and output y t+1 [0], find the difference dx = x between the current input and the previous input. t [0]-x t+1 [0], perform a linear calculation on it dy=Wdx, and then add the output from the previous step to get the complete result y. t[0]=dy+y t+1 [0]; For b1, input it into x t [1] Store in DRAM, and calculate the input difference dx=x between it and b0. t [1]-x t [0], then perform a linear calculation dy=Wdx, and then add the output of b0 to get the complete result y. t [1]=dy+y t [0], then output y t [1] Stored in DRAM;
[0052] When t is not equal to T, and Tt is even: For b1, load the input x from the previous step from DRAM. t+1 [1] and output y t+1 [1] Find the difference dx = x between the current input and the previous input. t [1]-x t+1 [1] Perform a linear calculation on it, dy=Wdx, and then add the output from the previous step to get the complete result y. t [1]=dy+y t+1 [1]; For b0, input it into x t [0] is stored in DRAM, and the input difference dx = x between it and b0 is calculated. t [0]-x t [1] Then perform a linear calculation dy=Wdx, and then add the output of b1 to get the complete result y. t [0]=dy+y t [1], then output y t [0] Stored in DRAM.
[0053] Output reference value decoding technology based on recalculation: For adjacent linear layers, the input of the next layer is no longer stored when calculating the inter-step difference. Instead, the input of the next layer is obtained by recalculating the activation function of the previous layer during the calculation, saving transmission bandwidth and transmission energy consumption. The specific application conditions and processing procedures are as follows:
[0054] When there are two linear layers l1 and l2 in the model, where the input of l2 is equal to the output of l1 calculated by activation functions (including ReLU, SiLU, GELU, etc.) (e.g., Up projection and Down projection in the Transformer structure); when performing the difference calculation, at a certain step t+1, it is necessary to store the input and output of l1 and l2 into DRAM, and in the subsequent step t, it is necessary to load the input and output of l1 and l2 from DRAM.
[0055] When the above situation occurs, the optimization process is performed: at step t+1, the input of l2 is no longer stored in DRAM; at step t, the input of l2 is no longer loaded from DRAM, but based on the output of l1, the input of l2 is recalculated through the activation function.
[0056] This embodiment provides a target generation system based on diffusion Transformer, which is feasible and includes:
[0057] The task requirement determination module is used to obtain task requirements, which include the generation requirements of images, videos and audio.
[0058] The target generation module is used to input the task requirements into the target generation model and output the corresponding generation results. The target generation model is built based on the diffusion Transformer model and consists of multiple stacked Transformer Blocks. Each Transformer Block contains a multi-head attention layer and a multi-layer perceptron layer. In the target generation process, the target generation model uses randomly generated Gaussian noise as the original input and combines time steps and the Classifier-free Guidance method to perform the content generation task.
[0059] In practice, this embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor runs the computer program to cause the electronic device to perform a target generation method based on diffusion Transformer as described above.
[0060] In practice, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned target generation method based on diffusion Transformer.
[0061] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A target generation method based on diffusion Transformer, characterized in that, include: Obtain task requirements, which include requirements for generating images, videos, and audio. The task requirements are input into the target generation model, and the corresponding generation results are output. The target generation model is built based on the diffusion Transformer model and consists of multiple stacked Transformer Blocks. Each Transformer Block contains a multi-head attention layer and a multi-layer perceptron layer. In the target generation process, the target generation model uses randomly generated Gaussian noise as the original input and combines time steps and the Classifier-free Guidance method to perform the content generation task.
2. The method according to claim 1, characterized in that, In the target generation process, the target generation model uses randomly generated Gaussian noise as the original input and performs multi-step denoising. Each denoising step requires the complete execution of the target generation model to obtain the final generation result. The multi-step denoising process is numbered T, T-1, ..., 1 in sequence. The number of the current denoising step is t. When t=T, the target generation model uses the original Gaussian noise and generation conditions as input. When t is not T, the target generation model uses the output result of step t-1 and generation conditions as input.
3. The method according to claim 2, characterized in that, During the target generation process, the Classifier-freeGuidance method is used to generate the content to be generated.
4. The method according to claim 3, characterized in that, The generation of the content to be generated using the Classifier-free Guidance method includes the following specific steps: For each generated object, each denoising process requires running two batches. One batch uses the category or text description of the generated object as the generation condition, and the other batch uses empty labels as the generation condition. The output of each step is obtained by linear extrapolation of the results of the two batches. Let the two batches be b0 and b1. For any linear layer, let the corresponding weight be W. Let the input corresponding to b0 and b1 at time t be x. t [0], x t [1], the output is denoted as y t [0], y t [1] The calculation process of the current linear layer is as follows: When t=T, for b0, input x t [0] Store in DRAM, then calculate the output as y t [0]=Wx t [0], y t [0] Store in DRAM; for b1, calculate the input difference dx=x between b1 and b0. t [1]-x t [0], then perform linear calculation dy=Wdx, and further combine the output of b0 to obtain the complete result y. t [1]=dy+y t [0]; When t is not equal to T, and Tt is odd: For b0, load the input x from the previous step from DRAM. t+1 [0] and output y t+1 [0], find the difference dx = x between the current input and the previous input. t [0]-x t+1 [0] Perform linear calculation dy=Wdx, and further combine it with the output of the previous step to obtain the complete result y. t [0]=dy+y t+1 [0]; For b1, input x t [1] Store in DRAM, calculate the input difference dx=x between b1 and b0. t [1]-x t [0], then perform linear calculation dy=Wdx, and further combine the output of b0 to obtain the complete result y. t [1]=dy+y t [0], then y t [1] Stored in DRAM; When t is not equal to T, and Tt is even: For b1, load the input x from the previous step from DRAM. t+1 [1] and output y t+1 [1] Find the difference dx = x between the current input and the previous input. t [1]-x t+1 [1] Perform linear calculation dy=Wdx, and further combine it with the previous output of b0 to obtain the complete result y. t [1]=dy+y t +1[1]; For b0, input it into x t [0] Store in DRAM, calculate the input difference dx = xt[0] - x t [1] Then perform linear calculation dy=Wdx, and further combine the output of b1 to obtain the complete result y. t [0]=dy+y t [1], then output y t [0] Stored in DRAM.
5. The method according to claim 4, characterized in that, When there are two linear layers l1 and l2, and at time t+1, the inputs and outputs of l1 and l2 need to be stored in DRAM, and in subsequent steps t, the inputs and outputs of l1 and l2 need to be loaded from DRAM, optimization is performed: at time t+1, the input of l2 is not stored in DRAM, and at time t, the input of l2 is not loaded from DRAM. Based on the output of l1, the input of l2 is recalculated through the activation function.
6. A target generation system based on diffusion Transformer, characterized in that, include: The task requirement determination module is used to obtain task requirements, which include the generation requirements of images, videos and audio. The target generation module is used to input the task requirements into the target generation model and output the corresponding generation results. The target generation model is built based on the diffusion Transformer model and consists of multiple stacked Transformer Blocks. Each Transformer Block contains a multi-head attention layer and a multi-layer perceptron layer. In the target generation process, the target generation model uses randomly generated Gaussian noise as the original input and combines time steps and the Classifier-free Guidance method to perform the content generation task.
7. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform a target generation method based on diffusion Transformer according to any one of claims 1-5.
8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements a target generation method based on diffusion Transformer as described in any one of claims 1-5.