Verified video model acceleration system and method based on dual sparsity

Abstract

The invention provides a text video model acceleration system and method based on double sparsity. The system comprises an inter-frame sequence compression module and a sparse full attention module. The inter-frame sequence compression module is configured to obtain a video frame sequence to be processed; determining a reference frame and a to-be-compressed frame based on the video frame sequence; calculating the target cosine similarity between each token in the to-be-compressed frame and the corresponding token or region in the reference frame; based on the target cosine similarity, the tokens in the reference frame and the to-be-compressed frame are combined; the sparse full attention module is configured to determine an attention matrix based on the merged video frame sequence; generating a binary block mask based on the attention matrix; and performing attention calculation on the attention matrix based on the binary block mask to obtain a calculation result, so as to solve the problems that the current T2V model is slow in reasoning speed and large in calculation resource consumption, and the model acceleration scheme cannot give consideration to both efficiency and quality.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a text-based video model acceleration system and method based on dual sparsity. Background Technology

[0002] With the rapid development of multimedia technology, T2V (Text-to-Video) generation technology has demonstrated enormous application potential in numerous fields such as film and television production, advertising, and virtual reality. In practical applications, to meet users' high demands for the detail and frame consistency of generated video footage, advanced T2V models have gradually abandoned spatiotemporal interleaved attention mechanisms, opting instead for a more computationally intensive full attention architecture. However, when generating high-resolution or long-duration videos, the length of the input video token sequence increases dramatically, posing a significant computational challenge to the full attention mechanism and severely restricting the practical application and deployment efficiency of T2V technology.

[0003] To address the aforementioned challenges and the massive computational burden of the full attention mechanism, current methods employ sequence compression (such as VidTome and FlightVGM) to reduce temporal redundancy between video frames. This involves merging or pruning similar tokens to shorten the sequence length, thereby reducing computational complexity. On the other hand, sparse attention methods are being developed to address the internal redundancy of attention computation itself. Some methods use fixed sparsity patterns, such as diagonal patterns, while others, although achieving dynamic sparsity, are primarily geared towards unimodal text sequences in Large Language Models (LLMs).

[0004] For sequence compression methods, most adopt fixed region comparison strategies or static compression ratios. This "one-size-fits-all" approach cannot adapt to the dynamic changes in inter-frame similarity of video content (such as changes in motion and stillness) and diffusion models at different denoising time steps, resulting in low compression efficiency or loss of key information, thus affecting the quality of the generated video. For sparse attention methods, fixed sparsity patterns cannot adapt to the diverse and dynamic sparse structures presented by different attention heads in full attention models; dynamic sparse methods designed for single-modal text sequences are not optimized for the multimodal sequence characteristics of "text tokens" and "video tokens" coexisting in T2V models, and fail to distinguish the importance of processing different modal tokens, which may damage the consistency between video and text prompts due to excessive sparsity. Summary of the Invention

[0005] This application provides a text-based video model acceleration system and method based on dual sparsity to solve the technical problems of slow inference speed, high computational resource consumption, and the inability of existing T2V model acceleration schemes to balance efficiency and quality.

[0006] The first aspect of this application provides a text-based video model acceleration system based on dual sparsity, comprising: The inter-frame sequence compression module and sparse full attention module are integrated into the Transformer module of the T2V model; The inter-frame sequence compression module is configured as follows: Obtain the video frame sequence to be processed; Based on the video frame sequence, a reference frame and a frame to be compressed are determined; the reference frame is one of the frames in the video frame sequence, and the frame to be compressed is a video frame in the video frame sequence other than the reference frame. Calculate the target cosine similarity between each token in the frame to be compressed and the corresponding token or region in the reference frame; Based on the target cosine similarity, the tokens in the reference frame and the frame to be compressed are merged. The sparse full attention module is configured as follows: The attention matrix is ​​determined based on the merged video frame sequence. Based on the attention matrix, a binary block mask is generated; Based on the binary block mask, attention calculation is performed on the attention matrix to obtain the calculation result.

[0007] In some embodiments, the inter-frame sequence compression module is further configured to: Calculate the target cosine similarity between two tokens that are spatially perfectly aligned with the reference frame in the frame to be compressed; Alternatively, calculate the cosine similarity between one of the tokens in the compressed frame and the corresponding position and nine tokens in the reference frame; Obtain the maximum cosine similarity and determine the target cosine similarity; Alternatively, calculate the target cosine similarity between one token in the compressed frame and all tokens in the reference frame.

[0008] In some embodiments, the inter-frame sequence compression module is further configured to: Determine whether the target cosine similarity is greater than a preset dynamic threshold; the preset dynamic threshold is determined by the denoising time step or attention head of the T2V model. If so, then similar token pairs are determined based on the tokens in the corresponding reference frame and the frame to be compressed; The similar token pairs are merged.

[0009] In some embodiments, the inter-frame sequence compression module is further configured to: Discard the similar tokens in the frame to be compressed, and retain the token of the reference frame; Alternatively, the similar token pairs can be averaged element by element. Alternatively, the maximum norm interpolation method can be used to merge the similar token pairs.

[0010] In some embodiments, the inter-frame sequence compression module is further configured to: The merged token is restored after sparse full attention is computed or before the video frame sequence enters the next Transformer module.

[0011] In some embodiments, the inter-frame sequence compression module is configured with an index table; the index table is used to store the original spatial location of the merged tokens and the location of the target tokens merged into the reference frame; The inter-frame sequence compression module is further configured to: After calculating sparse full attention or before the video frame sequence enters the next Transformer module, the index table is used to restore the merged token from the target token position to the original spatial position.

[0012] In some embodiments, the sparse full attention module is further configured to: The Q matrix and K matrix are divided into blocks of a preset volume along the length and dimensions of the video frame sequence; Perform mean pooling on all token vectors within each block to obtain compact blocks; Calculate the matrix product between the compact blocks to obtain the block attention score map; A binary block mask is generated based on the block attention score map.

[0013] In some embodiments, the token includes: a text token and an image token; The sparse full attention module is further configured as follows: Based on the block attention score map, sparse operations are performed to preserve the attention between the text token and the image token, and a binary block mask is generated based on the attention between the image tokens.

[0014] In some embodiments, the sparse full attention module is further configured to: Sort the blocks in each row of the K and V matrices from largest to smallest according to the binary block mask; A predetermined number of blocks at the top of the K matrix and the V matrix are determined, and attention calculation is performed on each block in the Q matrix to obtain the calculation result.

[0015] The second aspect of this application provides a method for accelerating text-based video models based on dual sparsity, applied to a text-based video model acceleration system based on dual sparsity as described in any one of the first aspects above, comprising: Obtain the video frame sequence to be processed; Based on the video frame sequence, a reference frame and a frame to be compressed are determined; the reference frame is one of the frames in the video frame sequence, and the frame to be compressed is a video frame in the video frame sequence other than the reference frame. Calculate the target cosine similarity between each token in the frame to be compressed and the corresponding token or region in the reference frame; Based on the target cosine similarity, the tokens in the reference frame and the frame to be compressed are merged. The attention matrix is ​​determined based on the merged video frame sequence. Based on the attention matrix, a binary block mask is generated; Based on the binary block mask, attention calculation is performed on the attention matrix to obtain the calculation result.

[0016] This application provides a text-based video model acceleration system and method based on dual sparsity. The system includes: an inter-frame sequence compression module and a sparse full attention module integrated into the Transformer module of a T2V model. The inter-frame sequence compression module is configured to: acquire a video frame sequence to be processed; determine a reference frame and a frame to be compressed based on the video frame sequence; the reference frame is one frame in the video frame sequence, and the frame to be compressed is the video frame in the video frame sequence excluding the reference frame; calculate the target cosine similarity between each token in the frame to be compressed and the corresponding token or region in the reference frame; and merge the tokens in the reference frame and the frame to be compressed based on the target cosine similarity. The sparse full attention module is configured to: determine an attention matrix based on the merged video frame sequence; generate a binary block mask based on the attention matrix; and perform attention calculation on the attention matrix based on the binary block mask to obtain the calculation result. This achieves the goal of improving the inference speed of the T2V model and reducing computational resource consumption through the text-based video model acceleration system, while balancing efficiency and quality. Attached Figure Description

[0017] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the runtime of the text-based video model acceleration system based on dual sparsity in this application.

[0019] Explanation of reference numerals in the attached figures: 1-Inter-frame sequence compression module; 2-Sparse full attention module. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0021] For example, current model acceleration methods, when applied to fully attention-based T2V models, suffer from the following significant drawbacks: 1. The huge computational bottleneck caused by full attention: The current T2V model has inherent flaws in its basic architecture. The computational complexity of the full attention mechanism is proportional to the square of the sequence length (O(N²), where N is the total length of the spatiotemporal sequence). This leads to a sharp increase in inference cost as the number of video frames (temporal dimension) or resolution (spatial dimension) increases, forming a performance bottleneck that is difficult to overcome.

[0022] 2. Current sequence compression methods have poor adaptability and are prone to damaging video quality: To address temporal redundancy between video frames, existing methods (such as VidTome and FlightVGM) attempt to shorten the sequence length by merging or pruning similar tokens. However, most of these methods employ fixed region comparison strategies (such as comparing only tokens at aligned positions) or static compression ratios. This "one-size-fits-all" approach cannot adapt to the dynamic changes in inter-frame similarity of video content (such as changes in motion and static content) and diffusion models at different denoising time steps, resulting in low compression efficiency or loss of key information.

[0023] 3. Existing sparse attention methods are not suitable for T2V models and ignore multimodal characteristics: Existing sparsification methods also have limitations regarding the internal redundancy of attention computation itself. Some methods use fixed sparsity patterns (such as diagonal patterns), and this static pattern cannot adapt to the diverse and dynamic sparse structures exhibited by different attention heads in full attention models. Other methods, although achieving dynamic sparsity, are mainly designed for unimodal text sequences in Large Language Models (LLMs) and have not been optimized for the multimodal sequence characteristics of T2V models where "text tokens" and "video tokens" coexist. They fail to distinguish the importance of processing different modal tokens and may compromise the consistency between video and text prompts due to excessive sparsity.

[0024] To address the technical problems of slow inference speed, high computational resource consumption, and the inability of model acceleration schemes to balance efficiency and quality in T2V models, this application provides a text-based video model acceleration system and method based on dual sparsity. The following describes the text-based video model acceleration system and method based on dual sparsity: To address the issues of slow inference speed, high computational resource consumption, and the inability of existing acceleration solutions to balance efficiency and quality in existing T2V models, this application proposes a system to accelerate text-to-video model inference by leveraging intra-frame and inter-frame sparsity. This system framework primarily comprises two core technology modules: an Inter-Frame Sequence Compression (IFSC) module and a Sparse Full Attention module. These two modules work collaboratively and are integrated into the Transformer module of the T2V model in a plug-and-play manner, executing dynamically during inference.

[0025] Its overall operation process is as follows: Figure 1 As shown: During the forward propagation of the Transformer block, the input video frame sequence is first compressed by the inter-frame sequence compression module 1 to shorten the sequence length; then, the compressed sequence is sent to the sparse full attention module 2 for efficient computation; finally, the output of the attention computation is restored to the original sequence length and structure through the decompression mechanism of the inter-frame sequence compression module 1, and then subsequent nonlinear operations and residual connections are performed.

[0026] The first aspect of this application provides a text-based video model acceleration system based on dual sparsity, comprising: The inter-frame sequence compression module 1 and the sparse full attention module 2 are integrated into the Transformer module of the T2V model; The inter-frame sequence compression module 1 is configured as follows: Obtain the video frame sequence to be processed; based on the video frame sequence, determine the reference frame and the frames to be compressed; the reference frame is one frame in the video frame sequence, and the frames to be compressed are all video frames in the video frame sequence except the reference frame; for the video frame sequence to be processed, select one frame as the reference frame and the other frames as the compressed frames. Measure the redundancy by calculating the cosine similarity between each token in the compressed frame and the corresponding token or region in the reference frame.

[0027] Calculate the target cosine similarity between each token in the frame to be compressed and the corresponding token or region in the reference frame.

[0028] Specifically, the inter-frame sequence compression module 1 is further configured as follows: Calculate the target cosine similarity between two tokens in the frame to be compressed and the reference frame that are spatially perfectly aligned; or calculate the cosine similarity between one token in the compressed frame and the corresponding position and nine tokens in the reference frame and their neighborhood; obtain the maximum cosine similarity and determine the target cosine similarity; or calculate the target cosine similarity between one token in the compressed frame and all tokens in the reference frame.

[0029] To adapt to video content in different scenarios (such as static backgrounds and dynamic objects), this application employs an adaptive multi-region comparison strategy. This strategy includes, but is not limited to: 1x1 alignment region comparison: Only compares two tokens in the frame to be compressed and the reference frame that are spatially perfectly aligned. This method has minimal computational overhead.

[0030] 3x3 Neighborhood Comparison: A token in the frame to be compressed is compared with its corresponding position in the reference frame and the nine tokens within its surrounding 3x3 neighborhood, and the token with the highest similarity is selected. This method can better handle small displacements in video frames.

[0031] Full-frame analysis and comparison: Compare a token in the frame to be compressed with all tokens in the reference frame. This method has the highest overhead, but it can capture non-local motion relationships in the video frame.

[0032] It is worth noting that the selection of the reference frame can also be adaptive, such as selecting the first frame of the sequence as the global reference, or selecting the immediately preceding frame as the local reference.

[0033] Based on the target cosine similarity, the tokens in the reference frame and the frame to be compressed are merged.

[0034] Specifically, the inter-frame sequence compression module 1 is further configured as follows: The algorithm determines whether the target cosine similarity is greater than a preset dynamic threshold. This preset dynamic threshold is determined by the denoising timestep or attention head of the T2V model. A merging operation is only performed when the similarity of a token pair exceeds a preset dynamic threshold (λ). This threshold is not fixed but can be dynamically adjusted based on different denoising timesteps or attention heads of the T2V model. Based on offline analysis, inter-frame similarity varies across different stages and modules; therefore, a dynamic threshold can better balance compression ratio and information loss.

[0035] If so, then based on the tokens in the corresponding reference frame and the frame to be compressed, similar token pairs are determined; and the similar token pairs are merged. By combining dynamic thresholds and various merging strategies, fine-grained video frame sequence compression control can be achieved.

[0036] Specifically, the inter-frame sequence compression module 1 is further configured as follows: Discard the similar tokens in the frame to be compressed and retain the tokens of the reference frame; or perform an element-wise averaging operation on the similar token pairs; or merge the similar token pairs using the maximum norm interpolation method.

[0037] Specifically, for token pairs deemed similar, this application provides several merging methods, including: Pruning: Discard similar tokens in the frame to be compressed, and keep the tokens in the reference frame unchanged.

[0038] Averaging: The feature vectors of similar token pairs are averaged element-wise. For each token, its feature vector is obtained, and the vectors of similar token pairs are added along their corresponding dimensions and then divided by 2 (or weighted average) to merge the similar token pairs.

[0039] Maximum-norm interpolation: This method uses maximum-norm interpolation to merge similar token pairs. It is a more advanced fusion strategy that can retain stronger feature signals while merging.

[0040] The inter-frame sequence compression module 1 is further configured to: The merged token is restored after sparse full attention is computed or before the video frame sequence enters the next Transformer module.

[0041] Specifically, the inter-frame sequence compression module 1 is configured with an index table; the index table is used to store the original spatial location of the merged tokens and the location of the target tokens in the reference frame into which they are merged. During the merging operation, the inter-frame sequence compression module 1 generates and caches an index table, which records the original spatial location of each merged token and which target token in the reference frame it is merged into.

[0042] The inter-frame sequence compression module 1 is further configured as follows: After calculating sparse full attention or before the video frame sequence enters the next Transformer module, the index table is used to restore the merged token from the target token position to the original spatial position.

[0043] Specifically, after the sparse full attention computation is completed and before entering the next Transformer block, the inter-frame sequence compression module 1 uses a cached index table to restore the sequence. Specifically, unmerged tokens remain in their original positions, while merged tokens are "copied" back from their corresponding positions in the reference frame using the index table, thus restoring the video frame sequence to its original length and spatiotemporal structure. This process ensures consistent input and output dimensions for each Transformer module, achieving seamless plug-and-play functionality.

[0044] The sparse full attention module 2 is configured as follows: Based on the merged video frame sequence, the attention matrix is ​​determined; based on the merged video frame sequence, the inter-frame temporal attention matrix is ​​calculated using a self-attention mechanism (such as scaling dot product attention in Transformer).

[0045] Based on the attention matrix, a binary block mask is generated.

[0046] Specifically, the sparse full attention module 2 is further configured as follows: The Q and K matrices are divided into blocks of a preset volume along the length and dimension of the video frame sequence; mean pooling is performed on all token vectors within each block to obtain compact blocks; matrix product between compact blocks is calculated to obtain a block attention score map; and binary block masks are generated based on the block attention score map.

[0047] Specifically, a sparse mask is dynamically predicted to determine which parts of the attention matrix can be skipped in computation. The specific process is as follows: 1. Divide the Query (Q) matrix and Key (K) matrix into blocks of fixed size along the sequence length dimension.

[0048] 2. Perform mean pooling on all token vectors within each block to obtain compact representations ˜Q and ˜K (i.e., compact blocks) for each block.

[0049] 3. Calculate the matrix product (˜Q·˜K) between the compact blocks. T This yields a low-resolution "block attention score map".

[0050] 4. Generate the final binary block mask based on this score map. For example, a preset number of blocks with the highest scores can be selected from each row as the parts to be calculated.

[0051] The tokens include: text tokens and image tokens; the sparse full attention module 2 is further configured as follows: Based on the block attention score map, sparse operations are performed to preserve the attention between the text token and the image token, and a binary block mask is generated based on the attention between the image tokens. When generating the binary block mask, all computations related to the image token are forcibly preserved. Specifically, in the binary block mask, blocks corresponding to "image token-image token" interactions and "image token-text token" interactions are always marked as "must be computed." This ensures that the model fully considers the guidance information of the text prompts at each frame of the generated video, maintaining a high degree of consistency between the generated content (images) and the text description (text).

[0052] Based on the binary block mask, attention calculation is performed on the attention matrix to obtain the calculation result.

[0053] Specifically, the sparse full attention module 2 is further configured as follows: Sort the blocks in each row of the K and V matrices from largest to smallest according to the binary block mask; determine the first preset number of blocks in the K and V matrices and perform attention calculation with each block in the Q matrix to obtain the calculation result.

[0054] Specifically, the generated binary block mask is applied to an efficient attention kernel (such as FlashAttention). During attention computation, the outer loop iterates through each block in the Q matrix, while the inner loop only loads and computes the blocks in the K and V matrices specified by the binary block mask. For blocks marked as "skipped," their corresponding matrix multiplication (Q... i K j T and P ij V j The computation will not be executed at all, thus achieving a significant acceleration in computation. To prevent numerical overflow caused by sparsity in online Softmax computation, this application can also integrate techniques such as SmoothK, which stabilizes the computation process by centering the K matrix and avoids quality problems such as texture corruption in the generated video.

[0055] This application provides a text-to-video model acceleration system based on dual sparsity. While effectively shortening the input sequence length and reducing the computational complexity of full attention, it maximizes the preservation of video temporal continuity and key details, avoiding quality degradation. It can also dynamically identify and skip unimportant attention calculations, thereby significantly reducing the amount of computation without affecting visual fidelity. By constructing a collaborative, efficient, and plug-and-play acceleration framework, it can be easily integrated and its inference performance can be significantly improved without retraining or fine-tuning existing T2V models. It achieves an inference speedup of 1.7 times or even higher while maintaining or almost eliminating the loss of video quality.

[0056] This application provides a video model acceleration system based on dual sparsity, addressing the significant computational bottleneck caused by full attention: This application reduces the length N of the video frame sequence involved in the computation from the source through an inter-frame sequence compression module 1. Since the complexity of full attention is O(N²), reducing the sequence length leads to a decrease in quadratic computation. Then, the sparse full attention module further sparsifies the shortened sequence, skipping a large number of matrix multiplications. This dual acceleration mechanism (reducing N + reducing N) ² The synergistic effect of computational terms (in the whole attention method) can overcome the computational bottleneck of full attention more effectively than a single method.

[0057] To address the problem that "existing sequence compression methods have poor adaptability and are prone to damaging video quality", the inter-frame sequence compression module 1 of this application adopts an adaptive multi-region comparison strategy, replacing the rigid comparison method in the prior art, enabling it to intelligently select the most suitable similarity measurement method according to the complexity of the video content.

[0058] To address the issue that existing sparse attention methods are not applicable to T2V models and ignore multimodal characteristics, this application's sparse full attention module 2 employs dynamic block mask prediction instead of a fixed sparse pattern, enabling it to adapt to the diverse and dynamic sparse structures presented by different attention heads in T2V models. Most importantly, this application solves the "multimodal blind spot" when existing sparse methods are directly applied to T2V models by forcibly retaining the computation of all text tokens. This ensures that text prompts provide continuous and effective guidance for the video generation process, thereby accelerating the process while maintaining a high degree of relevance and accuracy between the generated video content and the text description.

[0059] This application provides a text-based video model acceleration system based on dual sparsity, which has the following significant advantages: 1. Greater flexibility and accuracy: Most existing technologies employ fixed region comparison strategies (such as comparing only tokens with aligned positions). The adaptive multi-region comparison strategy adopted in this application (1x1 aligned region comparison, 3x3 neighborhood comparison, and full-frame analysis comparison) can flexibly select the most suitable comparison method according to the dynamic complexity of the video content (such as small displacements or large-scale movements), thereby more accurately identifying redundant tokens that can be merged.

[0060] 2. Improved Video Generation Quality: Existing technologies often employ a fixed compression ratio, which forcibly discards key information in stages where video details are rich or inter-frame variations are significant, leading to error accumulation and quality degradation. The dynamic threshold control mechanism in this application intelligently adjusts the compression intensity based on the actual inter-frame similarity of the model at different denoising time steps. While ensuring a high compression ratio, it effectively avoids the destruction of core video information, thereby generating higher-quality videos.

[0061] 3. Enhanced Adaptability: The compression strategy of this application is doubly adaptive, adapting to both the video content (through multi-region comparison) and the generation process (through dynamic thresholding). This makes the text-based video model acceleration method provided in this application more universal and robust than the static strategies of existing technologies, enabling it to determine a better "efficiency-quality" balance point under various text prompts and video scenarios.

[0062] 4. Easy integration and lossless compatibility: Through an index-preserving decompression mechanism, this application can losslessly restore the sequence to its original structure after attention calculation. This ensures that the entire compression-computation-decompression process is transparent to other modules of the Transformer, achieving true plug-and-play functionality without requiring modification or retraining of existing T2V models, significantly lowering the application threshold.

[0063] A second aspect of this application provides a method for accelerating text-based video models based on dual sparsity, applied to a text-based video model acceleration system based on dual sparsity described in any of the above embodiments, comprising: Obtain the video frame sequence to be processed; Based on the video frame sequence, a reference frame and a frame to be compressed are determined; the reference frame is one of the frames in the video frame sequence, and the frame to be compressed is a video frame in the video frame sequence other than the reference frame. Calculate the target cosine similarity between each token in the frame to be compressed and the corresponding token or region in the reference frame; Based on the target cosine similarity, the tokens in the reference frame and the frame to be compressed are merged. The attention matrix is ​​determined based on the merged video frame sequence. Based on the attention matrix, a binary block mask is generated; Based on the binary block mask, attention calculation is performed on the attention matrix to obtain the calculation result.

[0064] It is worth noting that the effects of the above method embodiments can be found in the effects of the above system embodiments, and will not be repeated here.

[0065] The above detailed embodiments further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above are merely specific embodiments of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A text-based video model acceleration system based on dual sparsity, characterized in that, include: The inter-frame sequence compression module (1) and the sparse full attention module (2) are integrated into the Transformer module of the T2V model. The inter-frame sequence compression module (1) is configured as follows: Obtain the video frame sequence to be processed; Based on the video frame sequence, a reference frame and a frame to be compressed are determined; the reference frame is one of the frames in the video frame sequence, and the frame to be compressed is a video frame in the video frame sequence other than the reference frame. Calculate the target cosine similarity between each token in the frame to be compressed and the corresponding token or region in the reference frame; Based on the target cosine similarity, the tokens in the reference frame and the frame to be compressed are merged. The sparse full attention module (2) is configured as follows: The attention matrix is ​​determined based on the merged video frame sequence. Based on the attention matrix, a binary block mask is generated; Based on the binary block mask, attention calculation is performed on the attention matrix to obtain the calculation result.

2. The text-based video model acceleration system based on dual sparsity according to claim 1, characterized in that, The inter-frame sequence compression module (1) is further configured as follows: Calculate the target cosine similarity between two tokens that are spatially perfectly aligned with the reference frame in the frame to be compressed; Alternatively, calculate the cosine similarity between one of the tokens in the compressed frame and the corresponding position and nine tokens in the reference frame; Obtain the maximum cosine similarity and determine the target cosine similarity; Alternatively, calculate the target cosine similarity between one token in the compressed frame and all tokens in the reference frame.

3. The text-based video model acceleration system based on dual sparsity according to claim 1, characterized in that, The inter-frame sequence compression module (1) is further configured as follows: Determine whether the target cosine similarity is greater than a preset dynamic threshold; the preset dynamic threshold is determined by the denoising time step or attention head of the T2V model. If so, then similar token pairs are determined based on the tokens in the corresponding reference frame and the frame to be compressed; The similar token pairs are merged.

4. The text-based video model acceleration system based on dual sparsity according to claim 1, characterized in that, The inter-frame sequence compression module (1) is further configured as follows: Discard the similar tokens in the frame to be compressed, and retain the token of the reference frame; Alternatively, the similar token pairs can be averaged element by element. Alternatively, the maximum norm interpolation method can be used to merge the similar token pairs.

5. The text-based video model acceleration system based on dual sparsity according to claim 1, characterized in that, The inter-frame sequence compression module (1) is further configured to: The merged token is restored after sparse full attention is computed or before the video frame sequence enters the next Transformer module.

6. The text-based video model acceleration system based on dual sparsity according to claim 5, characterized in that, The inter-frame sequence compression module (1) is configured with an index table; the index table is used to store the original spatial location of the merged token and the target token location merged into the reference frame; The inter-frame sequence compression module (1) is further configured as follows: After calculating sparse full attention or before the video frame sequence enters the next Transformer module, the index table is used to restore the merged token from the target token position to the original spatial position.

7. The text-based video model acceleration system based on dual sparsity according to claim 1, characterized in that, The sparse full attention module (2) is further configured as follows: The Q matrix and K matrix are divided into blocks of a preset volume along the length and dimensions of the video frame sequence; Perform mean pooling on all token vectors within each block to obtain compact blocks; Calculate the matrix product between the compact blocks to obtain the block attention score map; A binary block mask is generated based on the block attention score map.

8. The text-based video model acceleration system based on dual sparsity according to claim 7, characterized in that, The token includes: text tokens and image tokens; The sparse full attention module (2) is further configured as follows: Based on the block attention score map, sparse operations are performed to preserve the attention between the text token and the image token, and a binary block mask is generated based on the attention between the image tokens.

9. The text-based video model acceleration system based on dual sparsity according to claim 1, characterized in that, The sparse full attention module (2) is further configured as follows: Sort the blocks in each row of the K and V matrices from largest to smallest according to the binary block mask; A predetermined number of blocks at the top of the K matrix and the V matrix are determined, and attention calculation is performed on each block in the Q matrix to obtain the calculation result.

10. A method for accelerating text-based video models based on dual sparsity, applied to a text-based video model acceleration system based on dual sparsity as described in any one of claims 1 to 9, characterized in that, include: Obtain the video frame sequence to be processed; Based on the video frame sequence, the reference frame and the frame to be compressed are determined; The reference frame is one of the frames in the video frame sequence, and the frame to be compressed is a video frame in the video frame sequence other than the reference frame. Calculate the target cosine similarity between each token in the frame to be compressed and the corresponding token or region in the reference frame; Based on the target cosine similarity, the tokens in the reference frame and the frame to be compressed are merged. The attention matrix is ​​determined based on the merged video frame sequence. Based on the attention matrix, a binary block mask is generated; Based on the binary block mask, attention calculation is performed on the attention matrix to obtain the calculation result.

Images