Acceleration method and accelerator of video generation model using 3D attention mechanism

Abstract

The application discloses an acceleration method and an accelerator of a video generation model using a 3D attention mechanism, wherein the attention calculation process comprises the following steps: important parts in the attention calculation are checked by using a reduction speculation method, FP-FP is used to calculate matrix multiplication for the important parts, and FP-INT is used to calculate matrix multiplication for non-important parts; in the process of calculating the matrix multiplication by using the FP-INT method, a floating-point number mantissa and an integer product are obtained by a hybrid calculation engine through a table lookup method. Through the speculation-based similarity detection method and the cache lookup table architecture, the redundant operations of the attention calculation are significantly reduced, more than 65% of the high-overhead FP-FP calculation in the original attention calculation process is replaced by the low-overhead FP-INT calculation, the efficiency is improved, the energy consumption in the video generation process is reduced, and the large-scale video generation task is more economical and efficient.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method and accelerator for accelerating a video generation model that utilizes a 3D attention mechanism. Background Technology

[0002] Video generation models (VGMs) create realistic videos using diffusion models, demonstrating a deep understanding of physical laws and marking a milestone towards advanced world models. Among all VGMs, diffusion-based Transformer (DiT) models exhibit superior performance in video generation tasks, addressing the efficiency and scalability issues faced by traditional U-Net architectures. Early 2D+1D attention VGMs (2D+1D VGMs) processed spatial and temporal dimensions separately, but the generated videos exhibited temporal inconsistencies. In contrast, 3D attention-based VGMs (3D-VGMs) are a more powerful alternative, integrating spatial and temporal information to generate higher-quality, more consistent videos.

[0003] Despite their superior performance, 3D-VGMs require significantly more computational resources. Compared to 2D+1D VGMs, 3D-VGMs demand 2 to 4 times more computation in generating medium-length and long videos, respectively. Furthermore, unlike 2D+1D VGMs, the primary computational bottleneck in 3D-VGMs lies in the attention mechanism. Attention computation accounts for over 75% of the total video generation process.

[0004] Traditional video processing techniques have achieved significant compression effects in a wide variety of applications by leveraging the spatial and temporal similarities between image patches. For example, Discrete Cosine Transform (DCT) utilizes the similarities between images by transforming spatial data to the frequency domain, focusing most resources on low-frequency component computation, thereby reducing the processing of high-frequency details and minimizing data redundancy.

[0005] Based on this, we can analogously utilize similar methods to allow 3D-VGMs to benefit from the similarity between tokens. However, two challenges hinder the potential for accelerating 3D-VGMs inference.

[0006] 1) First, how to obtain similarity in real time and efficiently? Existing work obtains an approximate attention importance mapping through low-bit matrix multiplication, but this method results in an accuracy loss of more than 5% while providing almost no acceleration for 3D-VGMs.

[0007] 2) Secondly, how to maintain workload balance when similar tokens are randomly distributed? Although many studies reuse the results of one token for other similar tokens based on similarity, randomly distributed similar tokens lead to severe workload imbalance (computation utilization <37%). Summary of the Invention

[0008] In view of the above-mentioned deficiencies of the prior art, this invention provides an acceleration method and accelerator for video generation models utilizing a 3D attention mechanism. It employs a speculative similarity detection method to efficiently acquire similarity during the attention calculation process, and utilizes a cache lookup table-based computation engine to achieve efficient mixed-precision computation. Furthermore, it adopts an adaptive grouping strategy to address the uneven workload problem caused by randomly distributed similar tokens, thereby significantly improving the video generation efficiency of 3D-VGMs. To achieve the above objectives, the technical solution of this invention includes:

[0009] An acceleration method for video generation models utilizing 3D attention mechanisms includes the following process for attention computation: using reduction inference to examine important parts of the attention computation; using FP-FP to compute matrix multiplication for important parts; and using FP-INT to compute matrix multiplication for unimportant parts. During the FP-INT matrix multiplication process, the hybrid computing engine obtains the product of the floating-point mantissa and the integer by looking up a table.

[0010] A further improvement of the present invention is that the attention score P is calculated. c The process includes:

[0011] K is obtained by INT8 quantization of the K matrix. q Matrix and corresponding scaling factor;

[0012] Q is obtained by sparsifying the Q matrix using the similarity between tokens. c Matrix and index table;

[0013] The scaling factor and Q are obtained during the K-matrix quantization process. c Matrix multiplication;

[0014] Perform reduction prediction: sum the tokens corresponding to each frame in the K matrix and Q matrix respectively to obtain K. r Matrix and Q r Matrix, according to K r Matrix and Q r Matrix-based attention calculation yields a frame-level importance matrix, which is then used to identify important components within the matrix.

[0015] For the critical parts, Q is calculated using the FP-FP method. c×K T For non-critical parts, FP-INT method is used for calculation. The results are then combined to obtain the attention matrix;

[0016] The attention score P is obtained by performing a softmax operation on the attention matrix. c .

[0017] A further improvement of the present invention is that, according to K r Matrix and Q r The expression for calculating the importance matrix at the frame level is as follows:

[0018]

[0019] Where: t is the number of tokens contained in each frame.

[0020] A further improvement of the present invention is that the attention score P is calculated. c In the matrix, both the Q matrix and the K matrix include n frames, where n is a positive integer. The importance matrix of the frame level includes n... 2 Each of these values ​​serves as an importance index, representing the portion of the Q matrix corresponding to the K matrix. T The importance of multiplying corresponding frame parts in the matrix is ​​determined by the value. If the value is greater than a threshold, the product is considered important; otherwise, it is considered unimportant.

[0021] A further improvement of the present invention is that the process of calculating matrix A includes:

[0022] INT8 quantization is performed on matrix V to obtain matrix V. q and scaling factor;

[0023] Perform reduction prediction: Indicate the attention score P c The sum of each column in the matrix is ​​then multiplied by the result in the V matrix to obtain the column-level importance matrix; the important components are inferred from the column-level importance matrix.

[0024] For the important parts, P is calculated using the FP-FP method. c ×V; For non-critical parts, P is calculated using the FP-INT method. c ×V q ; and combine the results to obtain A c matrix;

[0025] The scaling factor obtained during the quantization of the V matrix is ​​compared with A. c Matrix multiplication;

[0026] Recovering A using the index table obtained during the Q-matrix sparsity process c The matrix is ​​a complete matrix A.

[0027] A further improvement of the present invention is that, in the process of calculating matrix A, the importance matrix of the column level is a one-dimensional vector, and each component is used to represent the importance of the corresponding column of matrix V in the multiplication process; when the component is greater than the threshold, the product of the corresponding column is regarded as the important part, otherwise it is regarded as the unimportant part.

[0028] A further improvement of this invention is that, in the process of calculating matrix multiplication using the FP-INT method, the floating-point form is FP16 and the integer form is INT8.

[0029] A further improvement of the present invention is that, in the process of calculating matrix multiplication using the FP-INT method, the lookup processing unit in the hybrid computing engine prefetches the corresponding product set from the lookup cache into the Table according to the mantissa of the floating-point number, and the lookup multiplier in the lookup processing unit obtains the corresponding product from the Table according to the integer, and operates on the exponent part of the floating-point number according to the corresponding product.

[0030] The present invention also provides an accelerator for a video generation model utilizing a 3D attention mechanism, for use in the above-described method, comprising a hybrid computing engine integrated on an accelerator chip, the hybrid computing engine being used to obtain the product of the mantissa of a floating-point number and an integer by means of a lookup table.

[0031] The technical solution provided by this invention has the following technical effects: This method significantly reduces redundant operations in attention computation by using a speculative similarity detection method and a cache lookup table architecture. It replaces over 65% of the high-overhead FP-FP computation in the original attention computation process with low-overhead FP-INT computation, thereby improving efficiency and reducing energy consumption in the video generation process, making large-scale video generation tasks more economical and efficient. Addressing the issue of high attention computation ratio (over 75%) in 3D-VGMs, the invention uses a speculative similarity detection method to efficiently and in real-time acquire similarity, and utilizes the similarity between tokens to sparsify the matrix in the attention computation process, reducing redundant computation. By designing a cache lookup table-based computation engine (SIMCORE), efficient FP-INT computation is achieved, avoiding bank conflicts and providing a new acceleration solution for hardware optimization of 3D-VGMs.

[0032] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0033] Figure 1 The diagram shows a computational unit based on a cache lookup table;

[0034] Figure 2This is a similarity detection method based on inference.

[0035] Figure 3 These are the results of the accuracy test.

[0036] Figure 4 To accelerate and improve energy efficiency assessment results. Detailed Implementation

[0037] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0038] like Figure 2 As shown, embodiments of the present invention provide an acceleration method for a video generation model utilizing a 3D attention mechanism. Important components (similarity) in the attention calculation are detected by calculating the importance matrix at the frame level, and the similarity between tokens is simultaneously used to sparsify the Q-matrix during the attention calculation process. This method mainly includes calculating the attention score P. c And calculate matrix A.

[0039] like Figure 2 As shown in (a), during the attention calculation process, the Q matrix and K matrix are calculated based on the input, and the attention score P is calculated based on the Q matrix and K matrix. c In this embodiment, the attention score P is calculated. c The process includes:

[0040] ① The K matrix is ​​obtained by performing INT8 quantization on the K matrix. q The matrix and its corresponding scaling factor. During the K-matrix quantization process, each column receives a scaling factor.

[0041] ② Use the similarity between tokens to sparse the Q matrix to obtain Q. c Matrix and index table. Figure 2 In the example, both the Q matrix and the K matrix correspond to two frames, each frame has three tokens, and each token corresponds to... Figure 2 (a) A row of squares in matrices Q and K. In this step, matrix sparsification is a prior art technique, and the process is as follows: Figure 2 As shown in (d).

[0042] ③ Obtain the scaling factor and Q during the quantization process of the K matrix. c Multiply corresponding columns of the matrix.

[0043] ④ Perform reduction and speculation: such as Figure 2 As shown in (b), K is obtained by summing the tokens corresponding to each frame in the K matrix and Q matrix respectively. r Matrix and Q r Matrix, according to K r Matrix and Q r Matrix-based attention calculations yield frame-level importance matrices, which are then used to identify important components. Based on K... r Matrix and Q r The expression for calculating the importance matrix at the frame level is as follows:

[0044]

[0045] Where t is the number of tokens contained in each frame. In this embodiment, t is 3.

[0046] Calculate attention score P c In the matrix, both the Q matrix and the K matrix include n frames (n is a positive integer), and the importance matrix of the frame level includes n... 2 Each of these values ​​serves as an importance index, representing the portion of the Q matrix corresponding to the K matrix. T The importance of multiplying corresponding frame parts in the matrix is ​​determined by the value. If the value is greater than a threshold, the product is considered important; otherwise, it is considered unimportant.

[0047] like Figure 2 As shown, in this embodiment, both the Q matrix and the K matrix include two frames; the importance matrix of the frame level includes four values, namely T1, T2, T3, and T4. T1 represents the part of the Q matrix corresponding to the first frame and the K matrix. T The importance of the first part of the multiplication; T2 represents the first frame corresponding part of the Q matrix and K. T The importance of multiplication in the second part.

[0048] ⑤ For important parts, calculate Q using the FP-FP method. c ×K T For non-critical parts, FP-INT method is used for calculation. The results are then combined to obtain the attention matrix;

[0049] ⑥ Perform a softmax operation on the attention matrix to obtain the attention score P. c .

[0050] The process of calculating matrix A includes:

[0051] ⑦ Perform INT8 quantization on matrix V to obtain matrix V. q And the scaling factor;

[0052] ⑧ Make regulatory inferences: such as Figure 2 As shown in (c), the attention score P c The sum of each column in the matrix is ​​then multiplied by the result in the V matrix to obtain the column-level importance matrix; the important components are inferred from the column-level importance matrix.

[0053] In the process of calculating matrix A, the importance matrix of the column levels is a one-dimensional vector, and each component is used to represent the importance of the corresponding column of matrix V in the multiplication process; when the component is greater than the threshold, the product of the corresponding column is regarded as the important part, otherwise it is regarded as the unimportant part.

[0054] ⑨ For important parts, use the FP-FP method to calculate P. c ×V; For non-critical parts, P is calculated using the FP-INT method. c ×V q ; and combine the results to obtain A c matrix;

[0055] ⑩ The scaling factor obtained during the quantization of the V matrix is ​​compared with that of A. c Matrix multiplication;

[0056] Recovering A using the index table obtained during the Q-matrix sparsity process c The matrix is ​​a complete matrix A.

[0057] Without compromising accuracy, over 65% of the FP-FP computations in the original attention calculation process are replaced by the less expensive FP-INT method.

[0058] like Figure 1 As shown, existing hardware architectures (such as GPUs) convert INT back to FP before calculating FP-INT operations, failing to achieve a substantial reduction in computational load while introducing additional computational overhead. To support high-performance FP-INT computation, this invention designs a Mixed Engine, an end-to-end computation engine based on a cache lookup table, which achieves efficient FP-INT computation while avoiding bank conflicts.

[0059] In the hybrid computing engine, the result of multiplying all FP16 mantissas with INT8 is stored in an on-chip lookup table buffer (LUT buffer). The hybrid computing engine also includes multiple lookup table processing units (LUT PEs), each with a corresponding lookup table. During each calculation, the result of multiplying the FP mantissa and INT is prefetched into the lookup table corresponding to each lookup table processing unit. Each lookup table processing unit quickly searches the corresponding lookup table for the multiplication result of the FP mantissa and INT to obtain the mantissa and exponent parts of the result, and calculates the exponent part (the calculation of the exponent part is an existing technique in the relevant floating-point specification) by shifting the multiplication unit. Through the two-level lookup table structure and the prefetching method before each calculation, this hybrid computing engine achieves efficient FP-INT calculation while avoiding memory conflicts.

[0060] Due to the random distribution of similar tokens, an imbalance in workload can occur during the calculation of the frame-level importance matrix. In some embodiments, an adaptive grouping strategy is employed to reduce workload imbalance without restoring sparsity, while seamlessly integrating the inference-based similarity detection method of this invention.

[0061] like Figure 3 As shown, this embodiment applies the inference-based similarity detection algorithm and adaptive grouping strategy to the attention calculation of 3D-VGMs, and compares the generated video with the original model without any optimization. We conducted accuracy tests on two state-of-the-art (SOTA) models, CogVideoX-5B and Open-Sora-Plan v1.2, based on CLIPSIM and VBench benchmarks. Figure 3 As shown, under these two evaluation metrics, the accelerator using the method of this embodiment does not suffer from any loss of accuracy, achieving lossless optimization of video generation.

[0062] Furthermore, we compared the performance and energy efficiency of the accelerator using the method in this embodiment with that of the NVIDIA A100 GPU (80G), two representative DiT-based methods (InterArch and CMC), and two representative methods for optimizing attention computation (Sanger and FACT). Figure 4 As shown, this invention achieves an average speedup of 5.21x and an energy efficiency improvement of 17.92x compared to the NVIDIA A100 GPU (80G). Furthermore, compared to current representative work, this invention achieves an average speedup of 1.45x and an energy efficiency improvement of 1.63x.

[0063] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. An acceleration method of a video generation model using a 3D attention mechanism, characterized in that, The process of performing attention calculation includes: using reduction speculation to examine the important parts of attention calculation; using FP-FP to calculate matrix multiplication for important parts; and using FP-INT to calculate matrix multiplication for unimportant parts. During the FP-INT matrix multiplication process, the hybrid computing engine obtains the product of the floating-point mantissa and the integer by looking up a table. Computing an attention score The process includes: INT8 quantized matrix matrix and corresponding scale factor​ Utilizing similarity between tokens to sparsify a matrix results in a matrix and an index table matrix and an index table Will The scaling factor is obtained during matrix quantization. Matrix multiplication; Perform specification speculation: respectively for Matrix and The sum of the tokens corresponding to each frame in the matrix is ​​obtained. Matrix and Matrix, according to Matrix and Matrix-based attention calculation yields a frame-level importance matrix, which is then used to identify important components within the matrix. For critical parts, FP-FP method is used for computation. For non-critical parts, FP-INT method is used for calculation. The results are then combined to obtain the attention matrix. Attention scores are obtained by performing a softmax operation on the attention matrix. ; calculate The matrix process includes: right INT8 quantization of the matrix yields the matrix and scaling factor; Perform reduction inference: adjust attention scores Sum each column in the table and combine the results with the sum of the columns in the table. Matrix multiplication yields a column-level importance matrix; the important components are inferred from the column-level importance matrix. For critical parts, FP-FP method is used for computation. For non-critical parts, FP-INT method is used for calculation. ; and combine the results to obtain matrix; Will The scaling factor obtained during matrix quantization and Matrix multiplication; use Index table recovery obtained during matrix sparsity process The matrix is ​​complete matrix; In the process of calculating matrix multiplication using the FP-INT method, the lookup processing unit in the hybrid computing engine prefetches the corresponding product set from the lookup cache into the Table based on the mantissa of the floating-point number. The lookup multiplier in the lookup processing unit retrieves the corresponding product from the Table based on the integer type and operates on the exponent part of the floating-point number based on the corresponding product.

2. The method for accelerating a video generation model using a 3D attention mechanism according to claim 1, characterized in that, according to Matrix and The expression for calculating the importance matrix at the frame level is as follows: ； in: It represents the number of tokens contained in each frame.

3. The method for accelerating a video generation model using a 3D attention mechanism according to claim 2, characterized in that, Calculate attention score middle, Matrix and The matrices all include Each frame For positive integers, the importance matrix at the frame level includes Each of the stated values ​​serves as an importance indicator, used to represent... The part of the corresponding frame in the matrix and The importance of multiplying corresponding frame parts in the matrix is ​​determined by the value. If the value is greater than a threshold, the product is considered important; otherwise, it is considered unimportant.

4. The method for accelerating a video generation model using a 3D attention mechanism according to claim 1, characterized in that, calculate In the matrix process, the importance matrix of column levels is a one-dimensional vector, and its components are used to represent... The importance of corresponding columns in the matrix during multiplication; when a component is greater than a threshold, the product of the corresponding columns is considered an important part, otherwise it is considered an unimportant part.

5. An acceleration method for a video generation model utilizing a 3D attention mechanism according to any one of claims 1 to 4, characterized in that, In the process of calculating matrix multiplication using the FP-INT method, the floating-point form is FP16 and the integer form is INT8.

6. An accelerator for a video generation model utilizing a 3D attention mechanism, used in the method described in any one of claims 1 to 5, characterized in that... It includes a hybrid computing engine integrated on an accelerator chip, which is used to obtain the product of the mantissa of a floating-point number and an integer by looking up a table.

Images