Attention mechanism dynamic sparsification and quantization method, system, device and medium

Abstract

The application discloses an attention mechanism dynamic sparseness and quantization method, system, device and medium, which are corresponding solutions, in the solutions: through a block granularity clustering algorithm, an input vector sequence is rearranged to obtain a vector sequence cluster of block granularity of aggregated semantic information, a clustering center of each cluster is taken as a representative element to calculate an attention score of each cluster, a clustering block with high importance is selected based on the attention score to obtain a block granularity sparse mask, in an attention calculation kernel, a vector sequence block that needs to be calculated is selectively read in according to the mask, then a block-by-block data smoothing operation is performed on the read-in vector sequence block, and symmetric quantization is performed. The above-mentioned solution obtains a hardware-friendly sparse structure beneficial to uniform task division and avoiding waste of calculation and bandwidth resources under the condition of ensuring model accuracy, thereby saving calculation and bandwidth resources, reducing the calculation amount of the attention mechanism module, and further improving the efficiency of video generation model reasoning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence video generation model inference acceleration, in particular to an attention mechanism dynamic sparsity and quantization method, system, device and medium. BACKGROUND

[0002] In recent years, video generation technology has made significant progress, among which the Diffusion Transformer (DiT) architecture that combines diffusion models and Transformer attention mechanisms has become the mainstream method. The video generation process of this architecture is essentially an iterative denoising process: the model starts from a random noise state and gradually predicts and removes noise through a Transformer-based neural network, ultimately generating clear and realistic video content. During the entire inference process, the attention mechanism, as the core component of DiT, has a computational complexity that is proportional to the square of the input sequence length. When processing high-resolution or long-duration videos, the sequence length increases significantly, and the computational overhead of the self-attention mechanism becomes the main bottleneck restricting the inference speed. Existing methods reduce computation through sparsity and quantization, but few jointly design both to accelerate the attention mechanism. Quantization methods reduce computation by quantizing the input matrix of the attention mechanism to more efficient low-bit data for computation. Sparse methods use the inherent sparsity of attention to skip unnecessary calculations, thereby reducing the amount of data actually participating in computation. According to the generation method of the sparse mask, sparse attention methods can be roughly divided into static sparse attention and dynamic sparse attention. Static sparse attention methods obtain a preset attention mask pattern based on empirical observations and assign a fixed type of sparse mask during computation. Dynamic sparse attention methods calculate and generate sparse masks in real time based on input data, adaptively adjusting according to the characteristics of the current data to focus more accurately on important information. Static attention methods are simple to implement, but lack adaptability to diverse sparse patterns, resulting in poor accuracy. Dynamic sparse attention methods are difficult to obtain hardware-friendly sparse structures that benefit task uniform division while avoiding waste of computational and bandwidth resources while ensuring model accuracy, and are also difficult to efficiently combine with quantization methods.

[0003] Therefore, the present application is proposed. SUMMARY

[0004] The application aims to provide an attention mechanism dynamic sparsity and quantization method, system, device and medium, which can obtain a hardware-friendly sparse structure that is beneficial to uniform task division and avoids waste of computing and bandwidth resources under the condition of ensuring model accuracy, can save computing and bandwidth resources, and efficiently combines with the quantization method to reduce the amount of calculation, thereby improving the efficiency of video generation model inference.

[0005] The application aims to achieve the above-mentioned purposes through the following technical solutions.

[0006] An attention mechanism dynamic sparsity and quantization method comprises the following steps:

[0007] The input vector sequence of the attention module in the video generation model is clustered in a block granularity, and rearranged according to the clustering result to obtain a vector sequence block with block granularity of aggregated semantic information; wherein the input vector sequence is obtained through linear transformation of the extracted image features, or is the output of the previous calculation module;

[0008] The mean center vector of the vector sequence block is used to calculate the attention score of the vector sequence block, and a part of the vector sequence block with the highest attention score is selected to construct a dynamic sparse mask;

[0009] The calculation tasks are divided, and the data corresponding to each calculation subtask obtained by the division are loaded into the on-chip memory in combination with the dynamic sparse mask;

[0010] The data loaded into the on-chip memory is subjected to a smoothing operation and a quantization process;

[0011] The attention operation is performed in combination with the data after the quantization process, and the output of the attention module is obtained through dequantization as the input of the next calculation module or the decoder; when serving as the input of the decoder, it is used to guide the decoder to generate consecutive video frames.

[0012] An attention mechanism dynamic sparsity and quantization system is used to implement the above-mentioned method, comprising:

[0013] A clustering and rearrangement unit is used to cluster the input vector sequence of the attention module in the video generation model in a block granularity, and rearrange according to the clustering result to obtain a vector sequence block with block granularity of aggregated semantic information; wherein the input vector sequence is obtained through linear transformation of the extracted image features, or is the output of the previous calculation module;

[0014] A dynamic sparse mask construction unit is used to calculate the attention score of the vector sequence block using the mean center vector of the vector sequence block, and to select a part of the vector sequence to construct a dynamic sparse mask;

[0015] a data loading unit, configured to divide a calculation task and load data to on-chip memory in combination with the dynamic sparse mask;

[0016] a data smoothing and quantization unit, configured to perform a smoothing operation on the data loaded to the on-chip memory and perform quantization processing;

[0017] an attention operation and dequantization unit, configured to perform attention operation in combination with the data after quantization processing and obtain an output of an attention module by dequantization as an input of a next-level calculation module or a decoder, when the output is an input of the decoder, the output is used to guide the decoder to generate a consecutive video frame.

[0018] A processing device, comprising: one or more processors; a memory configured to store one or more programs;

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the foregoing method.

[0020] A readable storage medium, storing a computer program, when the computer program is executed by a processor, the foregoing method is implemented.

[0021] As can be seen from the technical solutions of the present application, the hardware-friendly sparse structure that is beneficial to uniform task division and avoids waste of calculation and bandwidth resources can be obtained under the condition of ensuring model precision, and the calculation and bandwidth resources can be saved; moreover, the model inference efficiency can be improved by optimizing the attention mechanism module by dynamic sparsification and numerical quantization; meanwhile, the sparse technology and quantization technology are used in combination in the attention module, and the quantization precision is further optimized by using block-granularity data smoothing operation; therefore, the present application can reduce the calculation amount in the attention mechanism, and further accelerate the inference speed of the video generation model. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 A flowchart of an attention mechanism dynamic sparsification and quantization method provided by an embodiment of the present application.

[0024] Figure 2 A schematic diagram of an attention mechanism dynamic sparsification and quantization method provided by an embodiment of the present application.

[0025] Figure 3A schematic diagram of a dynamic attention mechanism sparsification and quantization system is provided for embodiments of the present application.

[0026] Figure 4 A schematic diagram of a processing device is provided for embodiments of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0028] First, the terms that can be used in the present text are described as follows:

[0029] The terms "comprise", "contain", "include", "have" or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, the inclusion of a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.) should be interpreted as not only including the explicitly listed technical feature element, but also including other technical feature elements that are not explicitly listed but are known in the art.

[0030] The term "consisting of" means excluding any technical feature element that is not explicitly listed. If this term is used in a claim, the term will make the claim closed, so that it does not contain technical feature elements other than the explicitly listed technical feature elements, except for conventional impurities related thereto. If the term only appears in a certain clause of the claim, it is only limited to the elements explicitly listed in that clause, and the elements described in other clauses are not excluded from the overall claim.

[0031] Unless otherwise explicitly specified or limited, the terms "mount", "connect", "connect", "fix", and the like should be broadly understood, for example: it can be fixedly connected, or it can be detachably connected, or it can be integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For a person of ordinary skill in the art, the specific meaning of the above terms in the present text can be understood according to the specific circumstances.

[0032] The following provides a detailed description of the dynamic sparsity and quantization method, system, device, and medium for attention mechanisms provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Unless otherwise specified in the embodiments of this invention, specific conditions are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all commercially available conventional products.

[0033] Example 1

[0034] This invention provides a method for dynamic sparsity and quantization of attention mechanisms, such as... Figure 1 As shown, it mainly includes the following steps:

[0035] Step 1: Perform block-level clustering on the input vector sequence of the attention module in the video generation model, and rearrange the sequence based on the clustering results to obtain a block-level vector sequence block that aggregates semantic information.

[0036] In this embodiment of the invention, the preferred implementation of performing block-level clustering on the input vector sequence of the attention module and rearranging it according to the clustering results is as follows: the input vector sequence includes: a query vector sequence, a key vector sequence, and a value vector sequence; wherein, the query vector sequence and the key vector sequence are respectively clustered at the block level and rearranged according to the corresponding clustering results; the value vector sequence is rearranged according to the rearrangement index of the key vector sequence.

[0037] In this embodiment of the invention, a preferred implementation method for block-level clustering is as follows: Clustering is performed iteratively, ensuring that the size of each cluster matches a preset block size; in each iteration, clustering operations are performed based on the number of clusters, and clusters that reach the block size are selected as complete clusters. The complete clusters are the vector sequence blocks that have completed clustering, and they are removed from the input vector sequence. The remaining vectors continue to the next iteration; this process is repeated until all vectors are assigned to the corresponding vector sequence blocks that have completed clustering; wherein, the number of clusters is calculated using the length of the input vector sequence and the block size (cluster block size), set as half the length of the input vector sequence divided by the block size, and the vector sequence blocks that have completed clustering are the clustering results that meet the block granularity requirements after clustering and selection.

[0038] For example, the clustering algorithm can be the K-means clustering algorithm.

[0039] Step 2: Calculate the attention score of the vector sequence block using the mean center vector of the vector sequence block, and select the vector sequence blocks with the highest attention scores to construct a dynamic sparse mask.

[0040] The preferred implementation method for this step is as follows:

[0041] (1) The obtained vector sequence block of the block granularity of the aggregated semantic information comprises: a query vector sequence block, a key vector sequence block, and a value vector sequence block.

[0042] (2) The mean center vector of each vector sequence block is selected as the representative element of the corresponding vector sequence block, and the representative elements selected from the query vector sequence block and the key vector sequence block are combined into corresponding tensors to obtain two tensors; the two tensors are subjected to matrix multiplication, and the attention score between each query vector sequence block and each key vector sequence block is obtained through a Softmax function (a normalization exponential function) to form an attention score matrix P.

[0043] (3) According to a preset sparse ratio, the S key vector sequence blocks with the highest attention scores are selected from the corresponding row of the attention score matrix for each query vector sequence block, and a dynamic sparse mask is constructed using all the query vector sequence blocks and the corresponding S key vector sequence blocks. Specifically, each row of the attention score matrix P represents a query vector sequence block, and each column represents a key vector sequence block; for each row in the attention score matrix P, the S key vector sequence blocks with the highest attention scores are selected, the mask value at the corresponding position is set to 1, and the mask value at the remaining positions is set to 0 to obtain a dynamic sparse mask.

[0044] Step 3, divide the calculation task, and load the data corresponding to each calculation subtask obtained by the division to the on-chip memory in combination with the dynamic sparse mask.

[0045] The preferred embodiment of this step is as follows: the calculation task is disassembled into mutually independent calculation subtasks, and the calculation subtasks are assigned to the streaming multi-processors of a GPU (graphics processing unit) for parallel calculation; wherein a calculation subtask grid composed of three dimensions of batch size, number of attention heads, and number of query vector sequence blocks (i.e. the number of query vector sequence blocks of the block granularity of the aggregated semantic information obtained after clustering and rearranging the input query vector sequence) is used to disassemble the task, and these homogeneous calculation subtasks are distributed to the streaming multi-processors of the GPU for processing; in a single calculation subtask, the streaming multi-processors load the batch size and the block granularity size of the query vector sequence blocks of the attention heads assigned from the high-bandwidth memory of the GPU to the on-chip memory, and then process the calculation subtask of the query vector sequence block ; wherein after loading the query vector sequence block to the on-chip memory, the block granularity mask corresponding to the query vector sequence block is loaded, and the corresponding key vector sequence block and value vector sequence block are loaded according to the block granularity mask; the query vector sequence block The corresponding block granularity mask is a corresponding row in the dynamic sparse mask, and only the position corresponding to the mask value of 1 is loaded The key vector sequence block and the value vector sequence block (since the value vector sequence is rearranged according to the rearrangement index of the key vector sequence, the position of the obtained value vector sequence block is corresponding to the key vector sequence block).

[0046] Step 4, smoothing operation is performed on the data loaded to the on-chip memory, and quantization processing is performed.

[0047] The preferred embodiment of the present step is as follows: the data loaded to the on-chip memory includes a query vector sequence block, a key vector sequence block and a value vector sequence block; the key vector sequence block and the value vector sequence block are subjected to smoothing operation; and the query vector sequence block and the smoothed key vector sequence block are subjected to symmetric quantization.

[0048] Step 5, attention operation is performed in combination with the data after quantization processing, and the output of the attention module is obtained through dequantization.

[0049] In the present step, the data after processing in the aforementioned step 4 is used for attention operation, and the operation process involved in this part can be referred to the conventional technology, which will not be described herein. Then, the final output is obtained through dequantization.

[0050] As understood by those skilled in the art, the input vector sequence in step 1 is obtained through linear transformation of the extracted image features (extracted from the input image), or is the output of the previous stage calculation module (for example, a multi-layer perception machine), and the final output can be used as the input of the next stage calculation module or decoder (to guide the subsequent decoder to generate more coherent video frames).

[0051] The above scheme provided by the embodiments of the present application optimizes the dynamic sparsity and numerical quantization of the attention mechanism module in the video generation model inference scene, thereby improving the model inference efficiency. Compared with the prior art, the present application uses the block granularity clustering method to perform semantic similar and task uniform division while avoiding the waste of computing and bandwidth resources, so that the data distribution is beneficial to the subsequent sparse and quantization optimization. Meanwhile, the present application combines the use of sparse technology and quantization technology in the attention mechanism, and further optimizes the quantization precision by using block granularity data smoothing operation. In combination with the above two points, the present application can reduce the calculation amount in the attention mechanism, and further accelerate the inference speed of the video generation model.

[0052] In order to more clearly show the technical solutions provided by the present application and the technical effects produced, the method provided by the embodiments of the present application is described in detail below with specific embodiments.

[0053] I. Overall introduction of the scheme.

[0054] In view of the fact that in a traditional scheme, token granularity clustering results in an uncertain number of tokens in each class, it is difficult to meet the requirement that the number of tokens processed at one time in a high-performance kernel can be divided by numbers such as 32 / 64 / 128, and the remainder part needs to be processed again using all thread resources, causing resource waste. Block granularity can well solve this problem. For example, if the block size is set to 32, the number of tokens in each class after clustering must be divisible by 32, thereby fully utilizing hardware resources. In addition, because the task is divided into block granularity, the amount of tasks allocated is also the same. Based on this, in the embodiment of the present application, the input vector sequence (query vector sequence, key vector sequence, value vector sequence) of the attention module is rearranged by a block granularity clustering algorithm (for example, a K-means clustering algorithm), and vector sequence clusters of block granularity of aggregated semantic information are obtained. The cluster center of each cluster is used as a representative element to calculate the attention score of each cluster, and based on the attention score, important clusters are selected to obtain dynamic sparsity. In the attention calculation kernel, the vector sequence block that needs to be calculated is selectively read according to the mask, and then the read vector sequence block is subjected to block-by-block data smoothing operation, and low-bit symmetric quantization is performed, thereby realizing efficient low-bit calculation. Thus, under the condition of ensuring the accuracy of the model, the computational complexity of the attention mechanism module is reduced, and the efficiency of the video generation model inference is improved.

[0055] II. Detailed introduction of the scheme.

[0056] As shown in Figure 2 , an example of the core process of the present application is shown.

[0057] 1. Cluster the input vector sequence of the attention module in block granularity, and rearrange according to the clustering result.

[0058] As shown in Figure 2 , step 1, the query vector sequence Q and the key vector sequence K input into the attention module are clustered in block granularity, Figure 2 In the figure, the rectangular boxes filled with different patterns represent different clustering clusters, and only four clustering clusters are provided as an example. In actual application, the number of clustering clusters can be set according to actual conditions.

[0059] In the embodiment of the present application, clustering is performed in an iterative manner to ensure that the size of each cluster is matched with the preset block size (for example, 64 or 128, etc.). In each round, a standard clustering (for example, K-means clustering) is performed with a smaller number of clusters (half of the sequence length / block size). After clustering is completed, complete clusters that have reached the block size are filtered out according to the cluster ID (identification) to which each vector belongs. These complete clusters are regarded as vector sequence blocks of completed clustering, and are temporarily removed from the input vector sequence for rearrangement. The remaining vectors in the input vector sequence continue to participate in the clustering and filtering process in the next round. This cycle continues until all vectors are allocated to the completed vector sequence blocks of matched size.

[0060] The input vector sequence introduced above mainly refers to the query vector sequence and the key vector sequence, both of which are clustered by using the above method. The value vector sequence V does not directly participate in clustering, but is synchronously rearranged according to the rearrangement index of the key vector sequence K. Through this step, a rearranged Q, K, V sequence is obtained, which are respectively organized into continuous and uniformly sized completed vector sequence blocks.

[0061] 2. Calculate the attention score of each vector sequence block using the mean center vector, and filter out the vector sequence blocks with high attention to obtain a sparse mask.

[0062] As Figure 2 In step 2, the query vector sequence Q and the key vector sequence K after clustering and rearrangement each form a completed vector sequence block arranged in block granularity order. The mean center vector of each vector sequence block is selected as the representative element of the vector sequence block. These representative elements form a small-scale tensor (i.e., a reduced tensor), which is denoted as The two reduced tensors are multiplied by matrix, and then the estimated attention score of the corresponding cluster block is obtained by the softmax function to form an attention score matrix P. For each query vector sequence block, the top-S key vector sequence blocks (the top-S key vector sequence blocks with the highest attention score) are retained, that is, the S key vector sequence blocks with the highest attention score are selected from the attention score matrix P for each query vector sequence block (corresponding to each row in the attention score matrix P). These selected blocks are used to construct a sparse mask, and the selected blocks are set to true (value 1) and the unselected blocks are set to false (value 0). Figure 2 In the example shown, the highest block in each row is selected to be true, i.e., S=1.

[0063] 3. Divide the calculation task grid and load data according to the sparse mask.

[0064] As Figure 2In step 3, the computing task is decomposed into a grid of independent sub-tasks, which are assigned to the Streaming Multiprocessor (SM) of the GPU, so as to realize efficient parallel computing. The task is decomposed using a sub-task grid composed of three dimensions of batch size, Attention Heads Num and the number of blocks of query vector sequence. The SM of the GPU is assigned to process a number of homogeneous sub-tasks. In a single computing sub-task, the SM loads the block granularity size of query vector sequence blocks of the assigned batch and attention head from the high-bandwidth memory HBM of the GPU to the on-chip memory, and processes the computing sub-task of the query vector sequence block. Then the query vector sequence block corresponding block granularity mask, which records whether the corresponding key vector sequence block and value vector sequence block participate in the operation. The SM only loads the key and value vector sequence blocks marked as True in the mask in turn to the on-chip memory, and skips those blocks marked as False. This way realizes sparse attention, significantly reducing the amount of calculation and memory bandwidth consumption. These processes correspond to the 1st, 2nd, 3rd, 4th, 6th and 7th rows of the subsequent Table 1 algorithm, where b is the block size, N is the number of blocks, L represents the sequence length, FP16 (Float16) represents 16-bit floating point number, i is the sequence number of the query vector sequence block, and j is the sequence number of the key vector sequence block and the value vector sequence block; the 1st row is the input (Input) of the algorithm; the 2nd row is the number of blocks calculated; the 3rd row defines the parallelized loop operation, which involves the subsequent 4th-14th rows; the 4th row is to load the query vector sequence block to the corresponding Streaming Multiprocessor; the 6th row defines the internal loop process, that is, for the query vector sequence block , in combination with the corresponding block granularity mask to determine whether the mask at the corresponding position j is true; the 7th row is to load the corresponding key and value vector sequence blocks when the mask at the position j is true .

[0065] 4. Data smoothing is performed on the loaded key and value vector sequence blocks, and the query, key and value vector sequence blocks are block-wise symmetrically quantized.

[0066] Data smoothing is performed on the key vector sequence and value vector sequence blocks. First, the mean vector of the key and value vector sequence blocks is calculated Then each key vector and value vector subtract the mean vector of the block, which makes the absolute value range of data distribution smaller and more concentrated around zero, thus improving the quantization accuracy. Symmetric quantization is performed on the query vector block and the smoothed key vector sequence block, which converts 16-bit floating-point number data into low-bit integer format (int8 or int4), obtaining low-bit query, key and value vector sequence blocks and the corresponding scaling factor These factors are used to restore the numerical range of the original data in the quantization and dequantization process. These processes correspond to lines 5, 8, 9 in Table 1 algorithm, Quantization means quantization is performed, is the corresponding quantization function, mean is the function of taking the average, is the mean vector, d is the dimension of the query vector sequence block Line 5 is the quantization of the query vector sequence block; line 8 is the calculation of the mean vector of the key and value vector sequence block, and then the corresponding subtraction of the mean vector is completed to smooth the data; line 9 is the quantization of the smoothed key and value vector sequence block.

[0067] 5. Perform low-bit matrix operation of attention mechanism and dequantization to obtain final output.

[0068] As Figure 2 Step 3, a series of calculations involved in the attention mechanism are performed, which corresponds to lines 10, 11, 12, 13, 14 in Table 1 algorithm. Figure 2 The sequence numbers on the right side of the third step , , Correspond. In Table 1, is the low-bit data, which is the quantized query vector sequence block, key vector sequence block, attention score matrix, value vector sequence block, on which the core operation matrix multiplication Matmul of the attention mechanism is performed, which can be efficiently executed by using the special hardware (Tensor Cores) of GPU. is an intermediate variable in the attention calculation process to ensure the correctness of the final result, where The corresponding represents the row-wise maximum function, row-wise sum function, diagonal matrix function, and matrix-vector multiplication function, respectively. Considering that a series of calculations involved in the attention mechanism can be understood with reference to conventional techniques, therefore, no further description is made. After loading and processing all the required key-value vector sequence blocks in the loop, the final output of the attention module is written to the video memory, and all the final outputs form a set O, which corresponds to lines 15, 16, 17, 18, 19 in Table 1 algorithm, where the subscript T represents transposition.

[0069] The above processes described in steps 3-5 are referred to as the implementation of per block sparse quant attention, and the specific process is shown in Table 1.

[0070] Table 1: Implementation of per block sparse quant attention

[0071]

[0072] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software and necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0073] Embodiment two

[0074] The present application also provides an attention mechanism dynamic sparsity and quantization system, which is mainly used to implement the method provided by the above-mentioned embodiments, as shown in the figure, the system mainly includes: Figure 3

[0075] A clustering and rearrangement unit is used to cluster the input vector sequence of the attention module in the video generation model at a block granularity, and rearrange according to the clustering result to obtain a vector sequence block of the block granularity of the aggregated semantic information; wherein the input vector sequence is obtained by linear transformation of the extracted image features, or is the output of the previous calculation module;

[0076] A dynamic sparse mask construction unit is used to calculate the attention score of the vector sequence block using the mean center vector of the vector sequence block, and to filter out a part of the vector sequence with the highest attention score to construct a dynamic sparse mask;

[0077] A data loading unit is used to divide the calculation task and load the data to the on-chip memory in combination with the dynamic sparse mask;

[0078] A data smoothing and quantization unit is used to perform smoothing operation on the data loaded to the on-chip memory and perform quantization processing;

[0079] ​The attention operation and dequantization unit is used to perform attention operations on the quantized data and dequantize it to obtain the output of the attention module, which is then used as the input to the next-level calculation module or the decoder. When used as the input to the decoder, it guides the decoder to generate coherent video frames.

[0080] Since the relevant processing details involved in this system have been described in detail in previous embodiments, they will not be repeated here.

[0081] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0082] Example 3

[0083] The present invention also provides a processing device, such as Figure 4 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0084] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0085] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0086] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0087] The output device can be a display terminal;

[0088] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0089] Example 4

[0090] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0091] The readable storage medium in the embodiments of the present application can be arranged in the foregoing processing device as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various media capable of storing program codes.

[0092] The above merely describes the preferred embodiments of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims. The information disclosed in the background section of the present application is merely intended to deepen the understanding of the general background of the present application, and should not be regarded as acknowledging or implying in any form that the information constitutes the prior art known by the person skilled in the art.

Claims

1. An attention mechanism dynamic sparsification and quantization method, characterized in that, The method comprises the following steps: performing block granularity clustering on an input vector sequence of an attention module in a video generation model, and rearranging according to the clustering result to obtain a vector sequence block of block granularity of aggregated semantic information; wherein the input vector sequence is obtained through linear transformation of extracted image features, or is the output of a previous calculation module; calculating the attention score of the vector sequence block using the mean center vector of the vector sequence block, and selecting a part of the vector sequence block with the highest attention score to construct a dynamic sparse mask; dividing the calculation task, and loading the data corresponding to each calculation subtask obtained by the division into on-chip memory in combination with the dynamic sparse mask; performing smoothing operation on the data loaded into the on-chip memory, and performing quantization processing; performing attention operation in combination with the data after quantization processing, and obtaining the output of the attention module through dequantization as the input of the next calculation module or the decoder, when the input of the decoder, used to guide the decoder to generate consecutive video frames.

2. The method of claim 1, wherein, The method comprises the following steps: The input vector sequence comprises a query vector sequence, a key vector sequence and a value vector sequence; The query vector sequence and the key vector sequence are respectively clustered in block granularity, and are respectively rearranged according to the corresponding clustering results; the value vector sequence is rearranged according to the rearrangement index of the key vector sequence.

3. The method of claim 1 or 2, wherein, The step of clustering in block granularity comprises: The clustering is performed in an iterative manner, so that the size of each clustering cluster matches the preset block size; In each iteration process, the clustering operation is performed with the number of clustering clusters, and the clustering cluster reaching the block size is selected as a complete clustering cluster, which is the vector sequence block after clustering, and is removed from the input vector sequence, and the remaining vectors continue to the next iteration; repeat constantly until all vectors are allocated to the corresponding complete clustering vector sequence block; The number of clustering clusters is calculated using the length of the input vector sequence and the block size.

4. The method of claim 1, wherein, The method comprises the following steps: The input vector sequence comprises a query vector sequence, a key vector sequence and a value vector sequence, and the obtained vector sequence block of block granularity of aggregated semantic information comprises a query vector sequence block, a key vector sequence block and a vector sequence block; The mean center vector of each vector sequence block is selected as the representative element of the corresponding vector sequence block, and the representative elements selected from the query vector sequence block and the key vector sequence block are combined into corresponding tensors to obtain two tensors; The two tensors are subjected to matrix multiplication, and the attention score between each query vector sequence block and each key vector sequence block is obtained through the Softmax function to form an attention score matrix P; wherein the Softmax function is a normalized exponential function. According to a preset sparse ratio, S key vector sequence blocks with the highest attention scores are selected from the corresponding row of the attention score matrix for each query vector sequence block, and a dynamic sparse mask is constructed using all the query vector sequence blocks and the corresponding S key vector sequence blocks.

5. The method of claim 4, wherein, The dynamic sparse mask is constructed using all the query vector sequence blocks and the corresponding S key vector sequence blocks. Each row of the attention score matrix P represents a query vector sequence block, and each column represents a key vector sequence block. For each row in the attention score matrix P, S key vector sequence blocks with the highest attention scores are selected, the mask values at the corresponding positions are set to 1, and the mask values at the remaining positions are set to 0, to obtain a dynamic sparse mask.

6. The method of claim 5, wherein, The data loading unit is configured to divide the calculation task and load data into the on-chip memory in combination with the dynamic sparse mask. The calculation task is divided into independent calculation sub-tasks, and the calculation sub-tasks are distributed to the streaming multi-processors of the GPU for parallel calculation; wherein, the calculation sub-task grid formed by the batch size, the number of attention heads, and the number of query vector sequence blocks is used to divide the task; and the GPU is a graphics processing unit; Within a single computational subtask, the streaming multiprocessor divides the query vector sequence into blocks of block granularity, allocated to batches and attention heads. The query vector sequence block is loaded from the GPU's high-bandwidth video memory and processed into on-chip memory. The computational subtask; in which, the query vector sequence block After being loaded into on-chip memory, the query vector sequence block is loaded. The corresponding block-level mask is used to load the corresponding key vector sequence block and value vector sequence block; this query vector sequence block... The corresponding block-granularity mask is the corresponding row in the dynamic sparse mask, and only the key vector sequence block and value vector sequence block corresponding to the position with a mask value of 1 are loaded.

7. The method of claim 1, wherein, The data smoothing and quantization unit is configured to perform smoothing operation on the data loaded into the on-chip memory and perform quantization processing. The data loaded into the on-chip memory includes query vector sequence blocks, key vector sequence blocks, and value vector sequence blocks; the key vector sequence blocks and the value vector sequence blocks are subjected to smoothing operation; and the query vector sequence blocks and the smoothed key vector sequence blocks are subjected to symmetric quantization.

8. An attention mechanism dynamic sparsification and quantization system, comprising: The method comprises the following steps: The clustering and rearranging unit is configured to cluster input vector sequences of an attention module in a video generation model at a block granularity, rearrange the input vector sequences according to the clustering result, and obtain vector sequence blocks with aggregated semantic information at a block granularity; wherein, the input vector sequences are obtained through linear transformation of extracted image features or are outputs of a previous calculation module; The dynamic sparse mask construction unit is configured to calculate attention scores of the vector sequence blocks using mean center vectors of the vector sequence blocks, and select a part of the vector sequence blocks with the highest attention scores to construct a dynamic sparse mask; The data loading unit is configured to divide the calculation task and load data into the on-chip memory in combination with the dynamic sparse mask; The data smoothing and quantization unit is configured to perform smoothing operation on the data loaded into the on-chip memory and perform quantization processing. The attention operation and dequantization unit is configured to perform attention operation in combination with the data after quantization processing, and obtain outputs of the attention module through dequantization, as inputs of a next calculation module or a decoder; when the outputs are used as inputs of the decoder, the outputs are used to guide the decoder to generate consecutive video frames.

9. A processing device, characterized by The method comprises the following steps: one or more processors; a memory configured to store one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-7.

10. A readable storage medium, storing a computer program, characterized in that, The computer program is executed by the processor to implement the method according to any one of claims 1-7.

Images