Video generation acceleration method for large model buffer optimization

By clustering and approximate calculation optimization of the Transformer model's attention calculation in video generation, the memory resource and computing efficiency problems are solved, and the speed and quality of video generation are improved.

CN119071590APending Publication Date: 2024-12-03PIO CLOUD COMPUTING (SHANGHAI) CO LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411126884.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In video generation, there are memory resource limitations and computing efficiency problems in the attention calculation process of the Transformer model, resulting in information loss and processing delays, affecting video quality and user experience.

Method used

By constructing and initializing the data structure to store cluster clusters and token values, clustering the tokens in the video stream in real time, retaining the key tokens and their cluster cluster information, and performing approximate calculations of matrix vector product and allocation functions. Optimize attention calculation formulas to reduce calculation burden.

Benefits of technology

It significantly reduces memory usage and computing time complexity, improves video generation speed and quality, and achieves acceleration of Transformer computing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure BDA0004996679360000021
    Figure BDA0004996679360000021
  • Figure BDA0004996679360000071
    Figure BDA0004996679360000071
  • Figure BDA0004996679360000091
    Figure BDA0004996679360000091
Patent Text Reader

Abstract

The invention relates to the technical field of computers, and provides a video generation acceleration method for large model buffer optimization. Comprising the following steps: constructing and initializing a data structure for storing a cluster and a corresponding token value; performing real-time clustering processing on the tokens extracted from the video stream, and distributing the tokens to corresponding clustering clusters according to features of the tokens; the key tokens in the clustering clusters are reserved and sampled based on a preset rule; and performing approximate calculation of a matrix vector product and a partition function based on the obtained data information to reduce the calculation burden during large model buffer optimization, and adjusting a clustering cluster and a corresponding token value in the data structure according to the approximate calculation result. According to the method, an online clustering method is adopted to compress a data structure, storage of redundant data is reduced, large-scale data flow processing is more efficient, and on the basis of an online clustering and sampling technology of key tokens, the time complexity of calculation is reduced, and the operation speed of an attention mechanism is increased.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a video generation acceleration method for large model buffer optimization. Background Art

[0002] In recent years, the generation of text to images has made significant progress, especially in diffusion models. The U-Net architecture has dominated static image generation due to its excellent performance and flexibility, and is widely used in large-scale generative models such as DALLE2 and Stable Diffusion. These models can generate high-quality images given text descriptions and are widely used in many fields such as art, advertising, entertainment, and education. However, with the increasing market demand for dynamic visual content, namely video generation, static image generation technology alone has gradually failed to meet the needs of current applications. Video generation requires not only image quality, but also the processing of temporal continuity and logical relationships between images. At this time, the Transformer-based model architecture has attracted attention due to its potential in processing sequence data. Unlike U-Net, which is mainly used for single-frame images, the Transformer architecture is proposed to handle the generation of video content through its advanced attention mechanism, especially in situations where temporal continuity and logical consistency need to be maintained. Therefore, the efficient computing requirements in video generation put higher demands on the computation of the Transformer, especially in the optimization of attention calculations.

[0003] In current video generation technology, especially when using the Transformer module for large-scale video generation tasks, there are the following problems:

[0004] (1) Memory resource limitation: A large amount of memory resources are required to store the key-value cache during the attention calculation process, but the memory resources of the computing nodes are extremely limited. In the actual calculation process, compression methods are often needed to reduce the required storage space. However, traditional compression methods may cause information loss, thus affecting the quality and accuracy of the generated video.

[0005] (2) Computational efficiency issues: Transformer’s attention calculation is essentially autoregressive, which means that key-value accumulation calculations need to be performed on all decoded embedded tokens during the calculation process. This not only increases the complexity and time consumption of the calculation, but also places higher demands on hardware resources. Especially in real-time or near-real-time application scenarios, this lengthy calculation process may cause processing delays, thereby reducing the system’s response speed and user experience. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to improve the speed and quality of video generation by efficiently compressing and optimizing the key-value cache, reducing memory usage and reducing the time complexity of attention calculation.

[0007] The above-mentioned object of the present invention is achieved through the following technical solutions:

[0008] The present invention provides a video generation acceleration method for large model buffer optimization, comprising:

[0009] Step S1, construct and initialize a data structure for storing clusters and corresponding token values;

[0010] Step S2, performing real-time clustering processing on the tokens extracted from the video stream, and assigning them to corresponding clusters according to their characteristics; retaining and sampling key tokens in the clusters based on preset rules, and storing the key tokens and the information of the clusters to which they belong in the data structure in step S1;

[0011] Step S3, based on the data information obtained in step S2, approximate calculations of matrix-vector products and partition functions are performed to reduce the computational burden during large model buffer optimization, and the clustering clusters and corresponding token values ​​in the data structure are adjusted according to the results of the approximate calculations.

[0012] Furthermore, step S1 also includes reconstructing the attention calculation formula based on the traditional attention calculation formula and data structure to optimize the calculation process of the matrix-vector product and the partition function; the attention calculation formula is,

[0013]

[0014] Attention(q n ,K n ,V n ) is the attention output, is the matrix-vector product; Σ i∈[n] exp( <k n ,q n >) is the partition function, q n is the query vector, K n :=[k1,k2,…,k n ], K n is the key vector matrix, consisting of key vectors k1, k2, …, k n Composition: V n :=[v1,v2,…,v n ],V n is the value vector matrix, corresponding to the key vector matrix, consisting of value vectors v1,v2,…,v n Composition, i is the index, n is the total number.

[0015] Furthermore, step S1 also includes that the error limit condition between the attention output and its overall estimated value in the sublinear memory is as follows:

[0016] ||z n -Attention(q n ,K n ,V n )||2≤ε||softmax(K n ,q n )||2||V n || op ;

[0017] z n is the overall estimate, Attention(q n ,K n ,V n ) is based on the given query vector q n , the key vector matrix K n Sum value vector matrix V n The calculated attention output, ||...||2 is the l2 norm, which is used to calculate the deviation between the overall estimate and the attention output; ε is a positive number, which is the allowable error threshold; softmax(K n ,q n ) is used to convert the dot product of the input key vector and query vector into a probability distribution; ||V n || op is the value vector matrix V n Operator norm of , used to measure the norm of the value vector matrix.

[0018] Further, step S2 is specifically:

[0019] Extract tokens from video streams in real time, analyze and extract the features of each token; and dynamically create or update clusters based on the features through clustering algorithms;

[0020] The preset rules including the highest frequency, the token closest to the cluster center or the uniqueness of the feature are used to calculate the importance score of each token, and the key tokens are selectively retained based on the importance score, and the information of the key tokens and their clusters are stored.

[0021] Furthermore, in step S3, the approximate calculation of the matrix-vector product is specifically,

[0022] Step S311, in the initial state, construct and set a data structure M containing an empty set of s empty values ​​for storing and updating key-value pairs, each key-value pair consisting of a key vector k and a corresponding value vector v;

[0023] Step S312, in each iteration of the video stream processing, dynamically updating the s key-value pairs in the data structure M based on the hit probability μ associated with each value vector v;

[0024] Step S313, calculate the matrix-vector product through the updated data structure M Approximate value of .

[0025] Further, step S312 is specifically,

[0026] Calculate the hit probability μ of the update operation, the calculation formula is,

[0027] Based on the calculated hit probability μ, the probability of selecting and updating s key-value pairs from the data structure M is, in, is the value vector v i The square of the l2 norm of .

[0028] Furthermore, in step S3, the approximate calculation of the partition function is performed in the following steps:

[0029] Step S321, initialize the constructed data structure D as an empty set, establish m empty clusters, and ensure that the clusters are mutually exclusive, and the distance between points in a single cluster does not exceed the threshold δ;

[0030] Step S322, when processing the tokens in the video stream, according to the characteristics of each token, select and assign it to the cluster corresponding to the nearest representative point, and adjust the point spacing within the cluster not to exceed the threshold δ;

[0031] Step S323: randomly sample t tokens from each cluster to form a corresponding subset for approximately calculating the partition function of the cluster.

[0032] Further, step S321 is specifically,

[0033] All tokens extracted from the video stream are taken as a point set, where each point represents the feature of a token;

[0034] Suppose that we have a positive integer m and a real number δ>0, and we transform the point set {x1,x2,…,x n} is clustered into (m,δ), that is, there are m clusters {C1,C2,…,C m}, and the clusters meet the following conditions: each cluster is independent and has no intersection; the union of all clusters covers the entire data set; the distance between any two points in the cluster does not exceed the threshold δ.

[0035] Further, step S322 is specifically,

[0036] For each newly formed cluster, select an initial element as a representative point. The representative point is the first token that flows in. During the continuous process of video stream processing, continuously calculate the distance between the newly flowing token and the representative point of each cluster to determine the classification of the token.

[0037] When a new token flows in, it is assigned to the corresponding cluster according to the distance from the existing representative point. Each cluster C i The size of i , that is, n i =|C i |;

[0038] For cluster C i Any two elements x i and k',||x i -k'||2≤δ; where x i is the representative point;

[0039] For any different clusters C i and C j The element x in i and x j ,||x i -x j ||2≥δ.

[0040] Further, step S323 is specifically,

[0041] From each cluster C i Randomly select t independent and identically distributed (iid) samples from the set to form a subset S i , used to approximate the clustering cluster C i The data structure D is dynamically updated according to the real-time data of the video stream to optimize the approximate calculation of the partition function.

[0042] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0043] The present invention significantly reduces memory usage by efficiently compressing the key-value cache. It mainly uses online clustering methods to compress data structures and reduce the storage of redundant data, making it more efficient when processing large-scale data streams. At the same time, online clustering and sampling technology based on key tokens reduces the time complexity of calculations, quickly processes key tokens, and accelerates the operation speed of the attention mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The present invention is a flowchart of the steps of the video generation acceleration method for large model buffer optimization. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0047] First embodiment

[0048] In existing video generation technologies, especially systems that generate video content through the Transformer model, the following problems exist in the attention calculation process: ① The key-value cache takes up a lot of memory space: During the attention calculation process, the memory of the computing node is limited, so the key-value cache usually needs to be compressed during the actual calculation process. ② The calculation process is lengthy and cumbersome: The original attention calculation is an autoregressive process, and the key-value accumulation calculation needs to be performed on all decoded embedded tokens during the calculation process.

[0049] In view of this, the present invention proposes a video generation acceleration method for large model buffer optimization, mainly targeting the large number of key-value caches generated in the attention calculation process during the video generation process, and proposes an efficient compression method to reduce memory usage and improve computing efficiency. This method mainly solves the above problems by performing efficient compression and clustering management on the key-value cache. The core idea of ​​this method is to reduce the amount of data that must be processed and stored through online clustering and selective key-value pair sampling, thereby significantly improving the computing speed and reducing memory usage without sacrificing the generation quality.

[0050] This embodiment optimizes attention calculation in video generation in the following aspects: ① Using online clustering technology, the input embedded tokens are dynamically clustered according to the features, and only the key tokens and their corresponding key-value pairs in each cluster are retained, which significantly reduces the storage requirements. ② The cluster clusters are updated and maintained in real time to ensure that each cluster only contains the most relevant tokens at the moment, further optimizing memory usage and computing resources. ③ Through a sampling strategy based on the design, the clustered key-value pairs are sampled and calculated, reducing the amount of data required for each calculation while ensuring the accuracy and efficiency of the calculation.

[0051] The specific operations of this embodiment are as follows: (1) Initialize data structure: construct and initialize a data structure for storing clusters and corresponding token values. (2) Online clustering and sampling: during the attention calculation process, cluster key tokens in real time, and partially retain and sample the values. (3) Approximate calculation and update: use a new data structure to approximate the output of the streaming attention mechanism, and dynamically update the clusters and token values ​​in the data structure. (4) Memory compression and computation optimization: through the above steps, efficient compression of the key-value cache and significant reduction in computational complexity are achieved, thereby improving the speed and quality of video generation.

[0052] The following are specific implementation methods: Figure 1 As shown, the present invention provides a video generation acceleration method for large model buffer optimization, comprising:

[0053] Step S1, construct and initialize a data structure for storing clusters and corresponding token values ​​for fast retrieval and update;

[0054] Step S2, performing real-time clustering processing on the tokens extracted from the video stream, and assigning them to corresponding clusters according to their characteristics; retaining and sampling key tokens in the clusters based on preset rules, and storing the key tokens and the information of the clusters to which they belong in the data structure in step S1;

[0055] Step S3, based on the data information obtained in step S2, approximate calculations of matrix-vector products and partition functions are performed to reduce the computational burden during large model buffer optimization, and the clustering clusters and corresponding token values ​​in the data structure are adjusted according to the results of the approximate calculations.

[0056] Through the new data structure, the output of the streaming attention mechanism is approximated, the amount of repeated calculations is reduced, and sublinear time complexity is achieved. Therefore, step S1 also includes an equivalent definition of attention calculation, and a reconstruction of the attention calculation formula based on the traditional attention calculation formula and data structure to optimize the calculation process of the matrix-vector product and the partition function; the attention calculation formula is,

[0057]

[0058] Attention(q n ,K n ,V n ) is the attention output, is the matrix-vector product; Σ i∈[n] exp( <k n ,q n >) is the partition function, q n is the query vector, K n :=[k1,k2,…,k n ], K n is the key vector matrix, consisting of key vectors k1, k2, …, k n Composition: V n :=[v1,v2,…,v n ],V n is the value vector matrix, corresponding to the key vector matrix, consisting of value vectors v1,v2,…,v n Composition, i is the index, n is the total number.

[0059] The attention decoding process is performed in a streaming manner, where multiple sequential embedding tokens form a vector triple sequence (q1, k1, v1), (q2, k2, v2), ..., (q n ,k n ,v n ), the attention calculation formula is Where K n :=[k1,k2,…,k n ],V n :=[v1,v2,…,v n ], the above attention calculation formula is equivalent to So in this application, we need to perform matrix-vector product and the partition function Σ i∈[n] exp ( <k n ,q n >) Reorganize the calculation method through a certain data structure to achieve overall calculation simplification.

[0061] Furthermore, step S1 also includes that the error limit condition between the attention output and its overall estimated value in the sublinear memory is as follows:

[0062] ||z n -Attention(q n ,K n ,V n )||2≤ε||softmax(K n ,qn )||2||V n || op ;

[0063] z n is the overall estimate, Attention(q n ,K n ,V n ) is based on the given query vector q n , the key vector matrix K n Sum value vector matrix V n The calculated attention output, ||...||2 is the l2 norm, which is used to calculate the deviation between the overall estimate and the attention output; ε is a positive number, which is the allowable error threshold; softmax(K n ,q n ) is used to convert the dot product of the input key vector and query vector into a probability distribution; ||V n || op is the value vector matrix V n Operator norm of , used to measure the norm of the value vector matrix.

[0064] Further, step S2 is specifically:

[0065] Extract tokens from video streams in real time, analyze and extract the features of each token; and dynamically create or update clusters based on the features through clustering algorithms;

[0066] The preset rules including the highest frequency, the token closest to the cluster center or the uniqueness of the feature are used to calculate the importance score of each token, and the key tokens are selectively retained based on the importance score, and the information of the key tokens and their clusters are stored.

[0067] In the matrix-vector product approximation algorithm based on standard sampling, when calculating the product A·B of matrix A and matrix B, the following steps are used:

[0068] First, we randomly sample an independent and identically distributed index i from the rows of matrix B. The probability of sampling is proportional to the l2 norm of the i-th row in matrix B. That is, the probability p of the i-th row being sampled is i It is given by the following formula:

[0069]

[0070] Among them B i,· represents the i-th row of matrix B, and ||·||2 represents the l2 norm.

[0071] Estimate matrix products: Estimate the value of A·B by multiplying the i-th column in A by the i-th row in B. That is, calculate the average of the following products:

[0072]

[0073] Where m is the number of sampling times, A ·,i represents the i-th column of matrix A, B i,· represents the i-th row of matrix B.

[0074] The standard sampling-based method can significantly reduce the complexity of matrix product calculations and is more efficient when processing large-scale matrix products. In traditional matrix product calculations, the computational complexity is O(d 2 ), where d is the dimension of the matrix. The above sampling method only requires O(ε -2 dlogn) samples can be added to the matrix The spectral norm of is guaranteed to contain only ε multiplication errors. In addition, since only part of the sampled data needs to be stored, the memory usage is greatly reduced, which is suitable for computing environments with limited memory. Moreover, the algorithm can control the error of the approximate result by selecting an appropriate number of sampling times m, thereby improving the calculation speed while ensuring the calculation accuracy.

[0075] Therefore, in the sampling process of step S3, its correctness is determined by introducing invariants that are maintained throughout the entire stream processing process. By constructing and maintaining a list of s elements represented by M, these elements are used in the calculation of the matrix-vector product. In each iteration of the data stream, s key-value pairs (kv) are updated in M, which belong to (k1,v1), (k2,v2), ..., (k n ,v n ), whose probability is proportional to . It is used to approximate the matrix-vector product, specifically,

[0076] Step S311, in the initial state, construct and set a data structure M containing an empty set of s empty values ​​for storing and updating key-value pairs, each key-value pair consisting of a key vector k and a corresponding value vector v;

[0077] Step S312, in each iteration of video stream processing, dynamically update the s key-value pairs in the data structure M based on the hit probability μ of the update operation associated with each value vector v; this probability ensures that the update of the key-value pairs follows the data distribution in the stream and maintains the correctness and efficiency of the algorithm.

[0078] Through the above steps, the key-value pair list M is dynamically maintained and updated when processing large-scale data streams, thereby achieving efficient matrix product calculation.

[0079] Step S313, calculate the matrix-vector product through the updated data structure M Approximate value of .

[0080] Further, step S312 is specifically,

[0081] Calculate the hit probability μ, the calculation formula is,

[0082] Based on the calculated hit probability μ, select and update s key-value pairs from the data structure M, that is, each element in M ​​is (k1,v1), (k2,v2),…, (k n ,v n The probability of an element in ) being independent and identically distributed is, in, is the value vector v i The square of the l2 norm of .

[0083] In the partition function Σ i∈[n] exp ( <k n ,q n >), a sampling and clustering-based method is used for sublinear time complexity approximate calculations. For any query, the partition function can be effectively approximated to a 1±∈ factor. In this application, the key can be covered by a sublinear number of bounded diameter clusters. Under this premise, a small number of uniformly sampled keys in each cluster can efficiently and accurately approximate the softmax normalized calculation. By designing a new data structure, the keys in the attention calculation are organized into a sublinear o(n) number of clusters, thereby greatly improving the computational efficiency and reducing memory usage. This calculation method provides greater flexibility for applications such as video generation that process large-scale data streams.

[0084] Furthermore, in step S3, the approximate calculation of the partition function is performed in the following steps:

[0085] Step S321, initialize the constructed data structure D as an empty set, establish m empty clusters, and ensure that the clusters are mutually exclusive, and the distance between points in a single cluster does not exceed the threshold δ. The steps are as follows:

[0086] All tokens extracted from the video stream are taken as a point set, where each point represents the feature of a token;

[0087] Suppose that we have a positive integer m and a real number δ>0, and we transform the point set {x1,x2,…,x n} is clustered into (m,δ), that is, there are m clusters {C1,C2,…,C m}, and the clusters meet the following conditions: each cluster is independent and has no intersection; the union of all clusters covers the entire data set; the distance between any two points in the cluster does not exceed the threshold δ. Specifically, the cluster meets the following conditions:

[0088] Mutual Exclusivity: For any two different clusters Ci and C j ,have

[0089] Coverage: The union of all clusters covers the entire point set, that is

[0090] Distance threshold: For any cluster C i For any two points x and y in , ||xy||2≤δ.

[0091] By constructing and maintaining a data structure represented by D, the partition function in the softmax denominator of any query is accurately approximated. The operations of the clustering process are as follows:

[0092] Step S322, when processing the tokens in the video stream, according to the characteristics of each token, select and assign it to the cluster corresponding to the nearest representative point, and adjust the point spacing within the cluster not to exceed the threshold δ;

[0093] Step S323: randomly sample t tokens from each cluster to form a corresponding subset for approximately calculating the partition function of the cluster.

[0094] Further, step S322 is specifically,

[0095] For each newly formed cluster, select an initial element as a representative point. The representative point is the first token that flows in. During the continuous process of video stream processing, continuously calculate the distance between the newly flowing token and the representative point of each cluster to determine the classification of the token.

[0096] Intra-cluster allocation: When a new token flows in, it is allocated to the corresponding cluster according to the distance from the existing representative point. Each cluster C i The size of i , that is, n i =|C i |;

[0097] Intra-cluster distance limit, for cluster C i Any two elements x i and k',||x i -k'||2≤δ; where x i is the representative point; when the element x i When the Euclidean distance of k' is less than a predetermined threshold δ, the similarity of the elements within the cluster is ensured.

[0098] Inter-cluster distance limit, for any different clusters C i and C j , where any element x iand x j ,||x i -x j ||2≥δ, when the element x i and x j The Euclidean distance of is greater than the predetermined threshold δ, ensuring the distinction between different clusters.

[0099] Further, step S323 is specifically,

[0100] From each cluster C i Randomly select t independent and identically distributed (iid) samples from the set to form a subset S i , used to approximate the clustering cluster C i The partition function is efficiently approximated to achieve sublinear memory complexity. At the same time, the data structure D is dynamically updated according to the real-time data of the video stream to ensure the accurate approximation of the softmax denominator partition function to optimize the approximate calculation of the partition function.

[0101] In summary, to address the problem of key-value cache occupying a large amount of memory space, this method proposes the following improvement measures:

[0102] (1) Online clustering and partial retention: Experience shows that tokens that initially have high attention scores will usually maintain high scores in the future decoding process, that is, the key tokens of the attention module have a significant clustering trend. Based on this observation, the present invention proposes to use online clustering for key tokens, sample the corresponding values ​​and partially retain them, so as to achieve sublinear memory usage. The specific method is as follows:

[0103] (2) Online clustering of key tokens: During the attention calculation process, key tokens are clustered in real time to form several clusters with significant clustering trends.

[0104] (3) Partial retention and sampling of values: For the token values ​​in each cluster, the partial retention and uniform sampling method is adopted to only retain the values ​​of some key tokens to reduce memory usage.

[0105] Through the above embodiments, the Transformer computing module in the video generation of the present invention realizes memory compression and computing speed improvement, significantly improves the generation speed of video frames, and improves the quality of generated videos. Specifically, it is embodied in:

[0106] (1) Online clustering and memory optimization: Experience and experimental results show that in the attention mechanism, tokens that initially have high attention scores usually also maintain high attention scores during the subsequent decoding process. This shows that there is a significant clustering trend among the key tokens in the attention module. Based on this observation, this method introduces online clustering technology. In the actual calculation process, the key tokens are clustered online in real time, and tokens with high similarity are grouped together. By sampling the values ​​of these clustering results and partially retaining these sampled values, the key-value cache is compressed. This strategy effectively reduces memory usage and achieves sublinear memory usage, thereby significantly reducing memory requirements without significantly affecting the generation quality.

[0107] (2) New data structure and computational optimization: The computational complexity of the attention mechanism is usually high, especially when processing long sequences or high-dimensional data, the amount of computation grows exponentially. Therefore, it is necessary to design more efficient data structures and computational methods to reduce the amount of computation and time complexity. This method designs a new data structure for approximate calculation of the output of the streaming attention mechanism. By performing reasonable approximation on the attention output, the amount of computation is significantly compressed. Specifically, the data structure can reduce unnecessary computational steps and data transmission while ensuring the accuracy of the calculation results, thereby achieving nonlinear time complexity. This not only improves computational efficiency, but also reduces the demand for hardware resources, enabling the system to complete high-quality video generation tasks in a shorter time.

[0108] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

[0109] It should be noted that the above embodiments can be freely combined as needed. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A video generation acceleration method for large model buffer optimization, characterized in that: include, Step S1, construct and initialize a data structure for storing clusters and corresponding token values; Step S2, performing real-time clustering processing on the tokens extracted from the video stream and assigning them to the corresponding clusters according to their characteristics; And retaining and sampling the key tokens in the clusters based on preset rules, and storing the key tokens and the information of the clusters to which they belong in the data structure in step S1; Step S3, based on the data information obtained in step S2, approximate calculation of matrix-vector product and partition function is performed to reduce the computational burden during large model buffer optimization, and the clustering clusters and the corresponding token values ​​in the data structure are adjusted according to the results of the approximate calculation.

2. The video generation acceleration method according to claim 1, characterized in that: Step S1 also includes reconstructing an attention calculation formula based on a traditional attention calculation formula and the data structure to optimize the calculation process of the matrix-vector product and the partition function; the attention calculation formula is: Attention(q n ,K n ,V n ) is the attention output, is the matrix-vector product; Σ i∈[n] exp( <k n ,q n >) is the partition function, q n is the query vector, K n :=[k1,k2,…,k n ], K n is the key vector matrix, consisting of key vectors k1, k2, …, k n Composition: V n :=[v1,v2,…,v n ],V n is a value vector matrix, the key vector matrix corresponds to the value vector v1,v2,…,v n Composition, i is the index, n is the total number.

3. The video generation acceleration method according to claim 2, characterized in that: Step S1 also includes that the error limiting condition between the attention output and its overall estimated value in the sublinear memory is as follows: ||z n -Attention(q n ,K n ,V n )||2≤ε||softmax(K n ,q n )||2||V n || op ; z n is the overall estimated value, Attention(q n ,K n ,V n ) is based on the given query vector q n , the key vector matrix K n and the value vector matrix V n The calculated attention output, ||...||2 is the l2 norm, which is used to calculate the deviation between the overall estimate and the attention output; ε is a positive number, which is the allowable error threshold; softmax(K n ,q n ) is used to convert the dot product of the input key vector and the query vector into a probability distribution; ||V n || op is the value vector matrix V n The operator norm of is used to measure the norm of the value vector matrix.

4. The video generation acceleration method according to claim 1, characterized in that: Step S2 is specifically: Extracting tokens from the video stream in real time, analyzing and extracting the features of each token; and dynamically creating or updating the clustering cluster according to the features through a clustering algorithm; The preset rules include the highest frequency, the token closest to the cluster center or the uniqueness of the feature, calculate the importance score of each token, selectively retain the key token based on the importance score, and store the key token and the information of the cluster to which it belongs.

5. The video generation acceleration method according to claim 3, characterized in that: In step S3, the approximate calculation of the matrix-vector product is specifically, Step S311, in an initial state, construct and set a data structure M containing an empty set of s empty values ​​for storing and updating key-value pairs, each of which consists of the key vector k and the corresponding value vector v; Step S312, in each iteration of the video stream processing, dynamically updating the s key-value pairs in the data structure M based on the hit probability μ associated with each of the value vectors v; Step S313, calculating the matrix-vector product through the updated data structure M Approximate value of .

6. The video generation acceleration method according to claim 5, characterized in that: Step S312 is specifically: Calculate the hit probability μ of the update operation, the calculation formula is, Based on the calculated hit probability μ, the probability of selecting and updating s key-value pairs from the data structure M is, i∈[n],j∈[s]; where is the value vector v i The square of the l2 norm of .

7. The video generation acceleration method according to claim 6, characterized in that: In step S3, the approximate calculation of the partition function is performed in the following steps: Step S321, initializing the constructed data structure D as an empty set, establishing m empty clusters, and ensuring that the clusters are mutually exclusive, and the distance between points in a single cluster does not exceed a threshold δ; Step S322, when processing the token in the video stream, according to the characteristics of each token, select and assign it to the cluster corresponding to the nearest representative point, and adjust the point spacing within the cluster not to exceed the threshold δ; Step S323: randomly sampling t tokens from each of the clusters to form a corresponding subset for approximately calculating the partition function of the clusters.

8. The video generation acceleration method according to claim 7, characterized in that: Step S321 is specifically: All the tokens extracted from the video stream are taken as a point set, each point representing a feature of the token; Assume that a positive integer m and a real number δ>0, and the point set {x1,x2,…,x n } is clustered into (m,δ), that is, there are m clusters {C1,C2,…,C m }, and the clusters satisfy the following conditions: each cluster is independent and has no intersection; the union of all clusters covers the entire data set; the distance between any two points in the cluster does not exceed the threshold δ.

9. The video generation acceleration method according to claim 8, characterized in that: Step S322 is specifically, For each newly formed cluster, an initial element is selected as the representative point, the representative point is the first token that flows in, and during the continuous process of the video stream processing, the distance between the newly flowing in token and the representative point of each cluster is continuously calculated to determine the classification of the token; When a new token flows in, it is assigned to the corresponding cluster according to the distance from the existing representative point. Each cluster C i The size of i , that is, n i =|C i |; For the cluster C i Any two elements x i and k',||x i -k'||2≤δ; wherein the x i is the representative point; For any different clusters C i and C j The element x in i and x j ,||x i -x j ||2≥δ.

10. The video generation acceleration method according to claim 9, characterized in that: Step S323, specifically, From each of the clusters C i Randomly select t independent and identically distributed (iid) samples from the set to form a subset S i , used to approximately calculate the cluster C i The distribution function is dynamically updated according to the real-time data of the video stream to optimize the approximate calculation of the distribution function.

Citation Information

Cited By

  • Acceleration method and accelerator for video generation model using 3D attention mechanism

    CN120547415A

  • Acceleration method and device for video generation model

    CN121217995A

  • Large language model-oriented adaptive KV cache compression method and system

    CN121542395A