Acceleration method and accelerator for video generation model using 3D attention mechanism

Abstract

The invention discloses an acceleration method and accelerator for a video generation model using a 3D attention mechanism, and the method comprises the following steps: employing a protocol speculation mode to check an important part in attention calculation, employing an FP-FP mode to calculate a matrix multiplication for the important part, and employing an FP-FP mode to calculate a matrix multiplication for the important part; for the non-important part, a matrix multiplication is calculated in an FP-INT mode; in the process of calculating the matrix multiplication by adopting an FP-INT mode, the hybrid calculation engine obtains the product of the mantissa and the shaping of the floating-point number in a table look-up mode. Through a speculation-based similarity detection method and a cache lookup table architecture, redundant operation of attention calculation is remarkably reduced, higher than 65% of high-overhead FP-FP calculation in the original attention calculation process is replaced with low-overhead FP-INT calculation, efficiency is approximately improved, energy consumption in the video generation process is reduced, and the video generation efficiency is improved. And the large-scale video generation task is more economical and efficient.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an acceleration method and accelerator for a video generation model using a 3D attention mechanism. Background Art

[0002] Video Generation Models (VGMs) demonstrate a deep understanding of physical laws by using diffusion models to create realistic videos, marking a milestone in the journey towards advanced world models. Among all VGMs, models based on the Diffusion Transformer (DiT) have shown superior performance in video generation tasks, solving the efficiency and scalability issues faced by traditional U-Net structures. The original 2D+1D attention VGMs (2D+1D VGMs) process the spatial and temporal dimensions separately, but the generated videos exhibit temporal inconsistencies. In comparison, 3D attention-based VGMs (3D-VGMs) are a more powerful alternative that can generate higher quality and more consistent videos by integrating spatial and temporal information.

[0003] Despite their superior performance, 3D-VGMs require significantly more computational resources. Compared to 2D+1D VGMs, 3D-VGMs require 2x to 4x more computation for medium-length video generation and long video generation, respectively. Furthermore, unlike 2D+1D VGMs, the primary computational bottleneck for 3D-VGMs shifts to the attention mechanism. Attention computation accounts for over 75% of the entire video generation process.

[0004] Traditional video processing techniques have achieved significant compression results in a variety of applications by exploiting the spatial and temporal similarities between image blocks. For example, the discrete cosine transform (DCT) exploits similarities between images by converting spatial data into the frequency domain, focusing most of the computation resources on low-frequency components, thereby reducing the processing of high-frequency details and minimizing data redundancy.

[0005] Based on this, we can analogously utilize similar methods for 3D-VGMs to benefit from the similarity between tokens. However, there are two challenges that hinder the potential of accelerating the inference of 3D-VGMs.

[0006] 1) First, how to obtain similarity efficiently in real time? Existing works obtain approximate attention importance maps through low-bit matrix multiplication, but this approach leads to more than 5% accuracy loss and hardly provides any speedup for 3D-VGMs.

[0007] 2) Secondly, how to maintain workload balance when similar tokens are randomly distributed? Although many studies reuse the results of a token for other similar tokens based on similarity, the random distribution of similar tokens leads to severe workload imbalance (computation utilization < 37%). Summary of the Invention

[0008] In view of the above-mentioned shortcomings of the prior art, the present invention provides an acceleration method and accelerator for a video generation model using a 3D attention mechanism. This method uses a speculative similarity detection method to efficiently obtain similarity during attention calculations, and utilizes a computational engine based on a cache lookup table to achieve efficient mixed-precision computations. Furthermore, an adaptive grouping strategy is employed to address the uneven workload caused by randomly distributed similar tokens, thereby significantly improving the video generation efficiency of 3D-VGMs. To achieve the above objectives, the technical solutions of the present invention include:

[0009] A method for accelerating a video generation model using a 3D attention mechanism includes the following steps: using a reduced inference method to verify important parts of the attention calculation, using an FP-FP method to calculate matrix multiplication for the important parts, and using an FP-INT method to calculate matrix multiplication for the unimportant parts; during the process of calculating matrix multiplication using the FP-INT method, a hybrid computing engine obtains the product of a floating-point number mantissa and an integer through a table lookup.

[0010] A further improvement of the present invention is to calculate the attention score P c The process includes:

[0011] K obtained by INT8 quantization of K matrix q matrices and corresponding scaling factors;

[0012] Using the similarity between tokens to sparsely calculate the Q matrix, we can get Q c matrices and index tables;

[0013] The scaling factor obtained during the quantization of the K matrix is ​​combined with Q c Matrix multiplication;

[0014] Perform reduction inference: sum the tokens corresponding to each frame in the K matrix and the Q matrix to get K r Matrix and Q r Matrix, according to K r Matrix and Q r Calculate attention matrix to obtain the importance matrix of frame level, and check the important parts of the matrix according to the importance matrix of frame level;

[0015] For important parts, Q is calculated using the FP-FP method. c×K T ; For non-important parts, use FP-INT method to calculate And combine the results to get the attention matrix;

[0016] The attention matrix performs Softmax operation to obtain the attention score P c .

[0017] A further improvement of the present invention is that according to K r Matrix and Q r The expression of the importance matrix for frame level calculation is:

[0018]

[0019] Where: t is the number of tokens contained in each frame.

[0020] A further improvement of the present invention is to calculate the attention score P c In the example, both the Q matrix and the K matrix include n frames, where n is a positive integer, and the importance matrix of the frame level includes n 2 Each value is used as an importance indicator to represent the part of the Q matrix corresponding to the frame and the K T The importance of the corresponding frame parts in the matrix when multiplied. When the value is greater than the threshold, the product is regarded as the important part, otherwise it is regarded as the unimportant part.

[0021] A further improvement of the present invention is that the process of calculating the A matrix includes:

[0022] Perform INT8 quantization on the V matrix to obtain the matrix V q and scaling factors;

[0023] Perform reduced inference: the attention score P c Sum each column in and multiply the result by the V matrix to obtain the column-level importance matrix; infer the important part based on the column-level importance matrix;

[0024] For the important part, FP-FP method is used to calculate P c ×V; for non-important parts, use FP-INT to calculate P c ×V q ; and combine the results to get A c matrix;

[0025] The scaling factor obtained during the quantization of the V matrix is ​​combined with A c Matrix multiplication;

[0026] Restore A using the index table obtained during the Q matrix sparse process c Matrix is ​​the complete A matrix.

[0027] A further improvement of the present invention is that, in the process of calculating the A matrix, the column-level importance matrix is ​​a one-dimensional vector, each component of which is used to represent the importance of the corresponding column of the V matrix in the multiplication process; when the component is greater than a threshold, the product of the corresponding column is regarded as an important part, otherwise it is regarded as an unimportant part.

[0028] A further improvement of the present invention is that, in the process of calculating matrix multiplication using the FP-INT method, the floating point number is in the form of FP16 and the integer is in the form of INT8.

[0029] A further improvement of the present invention is that, in the process of calculating matrix multiplication using the FP-INT method, the table lookup processing unit in the hybrid computing engine pre-fetches the corresponding product set from the table lookup cache to the Table according to the mantissa of the floating-point number, and the table lookup multiplier in the table lookup processing unit obtains the corresponding product from the Table according to the integer type, and operates the exponent part of the floating-point number according to the corresponding product.

[0030] The present invention also provides an accelerator for a video generation model utilizing a 3D attention mechanism, which is used to implement the above-mentioned method of the embodiment, and includes a hybrid computing engine integrated on the accelerator chip, wherein the hybrid computing engine is used to obtain the product of the mantissa of a floating-point number and an integer by looking up a table.

[0031] The technical solution provided by the present invention has the following technical effects: the method significantly reduces the redundant operations of attention calculation through a similarity detection method based on speculation and a cache lookup table architecture, and replaces more than 65% of the high-overhead FP-FP calculations in the original attention calculation process with low-overhead FP-INT calculations, which improves efficiency and reduces energy consumption in the video generation process, making large-scale video generation tasks more economical and efficient. In response to the problem that the proportion of attention calculation in 3D-VGMs is high (more than 75%), a similarity detection method based on speculation is used to obtain similarity in real time and efficiently, and the similarity between tokens is used to sparsely represent the matrix in the attention calculation process, thereby reducing redundant calculations. By designing a computing engine (SIMCORE) based on a cache lookup table, efficient FP-INT calculations are achieved, bank conflicts are avoided, and a new acceleration solution is provided for hardware optimization of 3D-VGMs.

[0032] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Shown is a computation unit based on a cache lookup table;

[0034] Figure 2For similarity detection based on speculation;

[0035] Figure 3 is the accuracy test result;

[0036] Figure 4 Evaluate results for acceleration and energy efficiency. DETAILED DESCRIPTION

[0037] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0038] like Figure 2 As shown, an embodiment of the present invention provides an acceleration method for a video generation model using a 3D attention mechanism. The important part (similarity) in the attention calculation is detected by calculating the importance matrix at the frame level, and the similarity between tokens is used to sparse the Q matrix in the attention calculation process. The method mainly includes calculating the attention score P c And calculate the A matrix.

[0039] like Figure 2 As shown in (a), during the attention calculation process, the Q matrix and the K matrix are calculated according to the input, and the attention score P is calculated based on the Q matrix and the K matrix. c In this embodiment, the attention score P is calculated c The process includes:

[0040] ① K obtained by INT8 quantization of K matrix q Matrix and the corresponding scale factor. During the K matrix quantization process, each column gets a scale factor.

[0041] ②Use the similarity between tokens to sparsely calculate the Q matrix and obtain Q c Matrices and index tables. Figure 2 In the example, the Q matrix and the K matrix correspond to two frames, each frame has three tokens, and each token corresponds to Figure 2 (a) A row of squares in the matrix Q and the matrix K. In this step, the sparsification of the matrix is ​​a prior art in the art, and the process is as follows Figure 2 (d) shown.

[0042] ③ The scaling factor and Q obtained in the process of quantizing the K matrix c Matrix multiplication is done column-wise.

[0043] ④ Carry out reduction and speculation: Figure 2 As shown in (b), the tokens corresponding to each frame in the K matrix and the Q matrix are summed to obtain K r Matrix and Q r Matrix, according to K r Matrix and Q r The attention matrix is ​​calculated to obtain the importance matrix of the frame level, and the important parts of the matrix are tested according to the importance matrix of the frame level. r Matrix and Q r The expression of the importance matrix for frame level calculation is:

[0044]

[0045] Where: t is the number of tokens contained in each frame. In this embodiment, t is 3.

[0046] Calculate the attention score P c In the example, both the Q matrix and the K matrix include n frames (n is a positive integer), and the importance matrix of the frame level includes n 2 Each value is used as an importance indicator to represent the part of the Q matrix corresponding to the frame and the K T The importance of the corresponding frame parts in the matrix when multiplied. When the value is greater than the threshold, the product is regarded as the important part, otherwise it is regarded as the unimportant part.

[0047] like Figure 2 As shown, in this embodiment, both the Q matrix and the K matrix include 2 frames; the importance matrix of the frame level includes four values, namely T1, T2, T3, and T4. T1 represents the first frame corresponding to the part in the Q matrix and the K matrix. T The importance of multiplying the first part in the Q matrix; T2 represents the first frame corresponding part and K T The importance of multiplication in the second part.

[0048] ⑤For important parts, use FP-FP method to calculate Q c ×K T ; For non-important parts, use FP-INT method to calculate And combine the results to get the attention matrix;

[0049] ⑥ Perform Softmax operation on the attention matrix to obtain the attention score P c .

[0050] The process of calculating the A matrix includes:

[0051] ⑦ Perform INT8 quantization on the V matrix to obtain the matrix V q and scale factor;

[0052] ⑧ Make specification inferences: Figure 2 As shown in (c), the attention score P c Sum each column in and multiply the result by the V matrix to obtain the column-level importance matrix; infer the important part based on the column-level importance matrix;

[0053] In the process of calculating the A matrix, the column-level importance matrix is ​​a one-dimensional vector, and its components are used to represent the importance of the corresponding column of the V matrix in the multiplication process; when the component is greater than the threshold, the product of the corresponding column is regarded as the important part, otherwise it is regarded as the unimportant part.

[0054] ⑨For the important part, use FP-FP method to calculate P c ×V; for non-important parts, use FP-INT to calculate P c ×V q ; and combine the results to get A c matrix;

[0055] ⑩Compare the scaling factor obtained during the quantization of the V matrix with A c Matrix multiplication;

[0056] Restore A using the index table obtained during the Q matrix sparse process c Matrix is ​​the complete A matrix.

[0057] Without losing accuracy, more than 65% of the FP-FP calculations in the original attention calculation process are replaced by the lower-overhead FP-INT method.

[0058] like Figure 1 As shown in the figure, considering that existing hardware architectures (such as GPUs) convert INT back into FP before calculating FP-INT operations, it cannot achieve a substantial reduction in the amount of computation but introduces additional computational overhead. To support high-performance FP-INT calculations, the present invention designs a mixed computing engine (Mixed Engine), an end-to-end computing engine based on cache lookup tables, which achieves efficient FP-INT calculations while avoiding bank conflicts.

[0059] In the hybrid computing engine, all the multiplication results of the mantissa of FP16 and INT8 are stored in the on-chip lookup table cache (LUT buffer). The hybrid computing engine also includes multiple lookup table processing units (LUT PE), each of which has a corresponding lookup table (Table). During each calculation process, the result of the multiplication of the FP mantissa and INT will be pre-fetched into the lookup table corresponding to each lookup processing unit in the figure. Each lookup processing unit quickly searches the mantissa part of FP and the multiplication result of INT in the corresponding lookup table to obtain the mantissa and exponent part of the result, and realizes the calculation of the exponent part by shifting the multiplication unit (the calculation of the exponent part is the existing technology in the relevant floating-point number specification). Through the two-level lookup table structure and the pre-fetching method before each calculation, the hybrid computing engine achieves efficient FP-INT calculation while avoiding storage conflicts.

[0060] Due to the random distribution of similar tokens, workload imbalance can occur during frame-level importance matrix calculation. In some embodiments, an adaptive grouping strategy is employed to reduce workload imbalance without restoring sparsity, while seamlessly integrating the inference-based similarity detection method of the present invention.

[0061] like Figure 3 As shown in the figure, the inference-based similarity detection algorithm and adaptive grouping strategy used in this embodiment are applied to the attention calculation of 3D-VGMs, and the generated video is compared with the original model without any optimization. We conducted accuracy tests on two state-of-the-art models, CogVideoX-5B and Open-Sora-Plan v1.2, based on the CLIPSIM and VBench benchmarks. Figure 3 As shown, under these two evaluation indicators, the accelerator using the method of this embodiment has no accuracy loss and achieves lossless optimization of video generation.

[0062] In addition, we compared the performance and energy efficiency of the accelerator using the method of this embodiment with NVIDIA A100 GPU (80G), two representative DiT-based methods, InterArch and CMC, and two representative methods for optimizing attention calculation, Sanger and FACT. Figure 4 As shown in the results, the present invention achieves an average speedup of 5.21 times and an energy efficiency improvement of 17.92 times compared to the NVIDIA A100 GPU (80G). At the same time, compared to the current representative work, the present invention achieves an average speedup of 1.45 times and an energy efficiency improvement of 1.63 times.

[0063] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A method for accelerating a video generation model using a 3D attention mechanism, characterized in that: The process of attention calculation includes: using the reduction inference method to check the important parts of the attention calculation, using the FP-FP method to calculate the matrix multiplication for the important parts, and using the FP-INT method to calculate the matrix multiplication for the unimportant parts; in the process of calculating the matrix multiplication using the FP-INT method, the hybrid computing engine obtains the product of the floating-point mantissa and the integer through table lookup.

2. The method for accelerating a video generation model using a 3D attention mechanism according to claim 1, wherein: Calculate the attention score P c The process includes: K obtained by INT8 quantization of K matrix q matrices and corresponding scaling factors; Using the similarity between tokens to sparsely calculate the Q matrix, we can get Q c matrices and index tables; The scaling factor obtained during the quantization of the K matrix is ​​combined with Q c Matrix multiplication; Perform reduction speculation: K q Matrix and Q c The sum of the Tokens corresponding to each frame in the matrix is ​​K r Matrix and Q r Matrix, calculate the attention according to the K matrix and the Q matrix, obtain the frame-level importance matrix, and test the important parts in the matrix according to the frame-level importance matrix; For important parts, Q is calculated using the FP-FP method. c ×K T ; For non-important parts, use FP-INT method to calculate And combine the results to get the attention matrix; The attention matrix performs Softmax operation to obtain the attention score P c .

3. The method for accelerating a video generation model using a 3D attention mechanism according to claim 2, wherein: According to K r Matrix and Q r The expression of the importance matrix for frame level calculation is: Where: t is the number of tokens contained in each frame.

4. The method for accelerating a video generation model using a 3D attention mechanism according to claim 3, wherein: Calculate the attention score P c In the example, both the Q matrix and the K matrix include n frames, where n is a positive integer, and the importance matrix of the frame level includes n 2 Each value is used as an importance indicator to represent the part of the Q matrix corresponding to the frame and the K T The importance of the corresponding frame parts in the matrix when multiplied. When the value is greater than the threshold, the product is regarded as the important part, otherwise it is regarded as the unimportant part.

5. The method for accelerating a video generation model using a 3D attention mechanism according to claim 2, wherein: The process of calculating the A matrix includes: Perform INT8 quantization on the V matrix to obtain the matrix V q and scaling factors; Perform reduced inference: the attention score P c Sum each column in and multiply the result by the V matrix to obtain the column-level importance matrix; infer the important part based on the column-level importance matrix; For the important part, FP-FP method is used to calculate P c ×V; for non-important parts, use FP-INT to calculate P c ×V q ; and combine the results to get A c matrix; The scaling factor obtained during the quantization of the V matrix is ​​combined with A c Matrix multiplication; Restore A using the index table obtained during the Q matrix sparse process c Matrix is ​​the complete A matrix.

6. The method for accelerating a video generation model using a 3D attention mechanism according to claim 5, wherein: In the process of calculating the A matrix, the column-level importance matrix is ​​a one-dimensional vector, and its components are used to represent the importance of the corresponding column of the V matrix in the multiplication process; when the component is greater than the threshold, the product of the corresponding column is regarded as the important part, otherwise it is regarded as the unimportant part.

7. The method for accelerating a video generation model using a 3D attention mechanism according to any one of claims 1 to 6, wherein: When calculating matrix multiplication using the FP-INT method, the floating-point number format is FP16 and the integer format is INT8.

8. The method for accelerating a video generation model using a 3D attention mechanism according to claim 1, wherein: In the process of calculating matrix multiplication using the FP-INT method, the table lookup processing unit in the hybrid computing engine pre-fetches the corresponding product set from the table lookup cache to the table according to the mantissa of the floating-point number. The table lookup multiplier in the table lookup processing unit obtains the corresponding product from the table according to the integer type, and operates the exponent part of the floating-point number according to the corresponding product.

9. An accelerator for a video generation model using a 3D attention mechanism, used in the method of any one of claims 1 to 8, characterized in that The invention comprises a hybrid computing engine integrated on an accelerator chip, wherein the hybrid computing engine is used for obtaining the product of a floating point number mantissa and an integer by looking up a table.

Images