Rapid video generation method and system based on structured space-time sparsity

Abstract

The invention discloses a quick video generation method and system based on structured space-time sparsity, and belongs to the field of computer vision and artificial intelligence. According to the method, an inherent structured and heterogeneous sparse mode of an attention matrix in a video diffusion model is systematically analyzed and revealed, and the core of the method is as follows: 1) a self-adaptive partitioning strategy is adopted, and various spatial interaction modes are approximated through dynamic image block grouping; 2) dynamically adjusting the sparse level of a window with time sequence change according to the inter-frame distance; and 3) automatically configuring a search algorithm, and optimizing a sparse mode while keeping a key attention path. The sparse attention mask optimized through offline pre-calculation is directly applied to attention calculation in a reasoning stage, so that 1.8-2.5 times of attention calculation acceleration is realized on a single GPU while the visual quality equivalent to a full attention baseline is kept, and a brand new solution path is provided for efficient long video generation.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and artificial intelligence, and in particular relates to a fast video generation method and system based on structured spatiotemporal sparsity. Background Art

[0002] In recent years, generative models have made significant progress in high-quality video synthesis. However, processing extremely long video sequences remains a key bottleneck. For Transformer-based video generation models, the core self-attention mechanism has a quadratic computational complexity, which becomes a fundamental challenge when processing video sequences containing massive spatiotemporal tokens. For example, in certain high-definition video generation tasks, generating a video of several seconds may involve processing more than 100,000 tokens, with attention calculations accounting for the vast majority of the total generation time. This enormous computational burden has severely restricted the development and application of long video generation technology.

[0003] Existing research shows that in video generation tasks, the full attention matrix is ​​actually highly sparse, which means there is a huge, untapped potential for acceleration. The challenge lies in how to effectively exploit these complex, structured, but seemingly irregular sparsity patterns. Existing sparse attention methods, such as fixed patterns (e.g., diagonal, blocky patterns) or dynamic prediction patterns, have limitations when applied to video generation. Fixed patterns cannot adapt to the diverse behavior of video content and attention heads, while dynamic prediction methods (determining which positions are sparse at runtime) introduce additional computational overhead, often offsetting the speed gains brought by sparse computation. In particular, the inherent three-dimensional (temporal, height, width) redundancy in video data poses a challenge to the two-dimensional sparse attention methods used in direct transfer language models. Therefore, there is an urgent need for an efficient attention computation method specifically for video generation tasks that can adaptively capture and exploit dynamic spatiotemporal sparsity patterns. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem of huge computational overhead in the self-attention mechanism of the existing Transformer-based video generation model due to processing ultra-long sequences, and to provide a fast video generation method and system based on structured spatiotemporal sparsity.

[0005] In order to achieve the above-mentioned object of the invention, the present invention specifically adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a method for fast video generation based on structured spatiotemporal sparsity, which comprises the following steps:

[0007] S1. Offline analysis of the pre-trained video generation model to obtain the attention map generated by its internal self-attention module when processing video data;

[0008] S2. Identify stable, structured spatiotemporal sparse patterns in the attention map, where the spatiotemporal sparse patterns include at least different patterns in spatial and temporal dimensions; wherein the patterns in spatial dimensions are local patterns, cross patterns, and global patterns, and the patterns in temporal dimensions are time-varying patterns and time-invariant patterns;

[0009] S3. Based on the identified spatiotemporal sparse patterns, an offline mask search method automatically searches for the optimal sparse attention mask configuration for each attention head of the video generation model to form a sparse attention mask library;

[0010] S4. During the online inference phase of the video generation model, according to the currently processed attention head and the frame group to which it belongs, the corresponding pre-computed sparse attention mask is loaded from the sparse attention mask library and applied to the self-attention calculation to skip the calculation of non-critical areas, thereby achieving acceleration.

[0011] Based on the above solution, each step can be implemented in the following preferred specific manner.

[0012] As a preferred embodiment of the first aspect above, in step S1, a pre-trained video generation model is first analyzed offline, and a series of representative text prompts are input to drive the video generation model to generate videos. During the video generation process, the complete attention map of each attention head in all Transformer layers within the video generation model is captured and saved.

[0013] As a preferred embodiment of the first aspect above, in step S3, the design of the sparse attention mask adopts a dual attention window strategy.

[0014] As a preferred embodiment of the first aspect, in step S3, the specific process of searching for the sparse attention mask configuration using the offline mask search method is as follows:

[0015] S31. In each attention head of the video generation model, divide the video data into multiple different frame groups, each frame group contains the same number of frames, and the number of frame groups in each attention head is the same;

[0016] S32. For an attention head, initialize a mask covering the area where the entire attention map is located in each frame group, preset a minimum recall rate threshold and a maximum computational cost threshold, continuously iterate to shrink the mask boundary, and preferentially remove areas whose attention contribution does not reach the preset target in each iteration. When the attention recall rate of the mask is lower than the minimum recall rate threshold or the computational cost gain of further shrinkage is lower than the maximum computational cost threshold, the iteration terminates and outputs the sparse attention mask configuration of the attention head;

[0017] S33: Change the text prompt or random seed used, regenerate the sparse attention mask configuration of the attention head according to step S32, and then merge the sparse attention mask configurations of the attention head to obtain the optimal sparse attention mask configuration of the attention head;

[0018] S34. Process all attention heads of the video generation model according to steps S32 to S33, and form a final universal sparse attention mask library by configuring all optimal sparse attention masks.

[0019] As a preferred embodiment of the first aspect, in step S33, the merging operation is a union operation.

[0020] As a preferred embodiment of the above-mentioned first aspect, in step S4, for each self-attention calculation in the video generation model, first, according to the layer and attention head in which it is located, the window size of the corresponding indication mask is directly loaded from the offline generated sparse attention mask library, and then the self-attention calculation is performed based on the hardware-aware tile method, and the query tensor, key tensor and value tensor are each divided into tiles in the sequence dimension. For a row vector of the query tensor tile, the column vector participating in the calculation in the key tensor is obtained through the sparse attention mask, and the row vector is dot-producted with the column vector participating in the calculation to calculate the attention score, so as to maintain the regularity of the calculation and improve hardware efficiency.

[0021] As a preferred embodiment of the first aspect above, in step S4, the video generation model adopts the DiT architecture and reuses the same set of sparse attention masks in multiple consecutive denoising steps to reduce the frequency of mask loading.

[0022] In a second aspect, the present invention provides a fast video generation system based on structured spatiotemporal sparsity, comprising:

[0023] The attention map generation module is used to analyze the pre-trained video generation model offline and obtain the attention map generated by the internal self-attention module when processing video data;

[0024] a sparse pattern recognition module for identifying stable, structured spatiotemporal sparse patterns in the attention map, wherein the spatiotemporal sparse patterns include at least different patterns in the spatial dimension and the temporal dimension; wherein the patterns in the spatial dimension are local patterns, cross patterns, and global patterns, and the patterns in the temporal dimension are time-varying patterns and time-invariant patterns;

[0025] A mask library acquisition module is used to automatically search for the optimal sparse attention mask configuration for each attention head of the video generation model based on the identified spatiotemporal sparse patterns using an offline mask search method to form a sparse attention mask library;

[0026] The online inference module is used to load the corresponding pre-computed sparse attention mask from the sparse attention mask library according to the currently processed attention head and the frame group to which it belongs during the online inference stage of the video generation model, and apply it to the self-attention calculation to skip the calculation of non-critical areas, thereby achieving acceleration.

[0027] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for fast video generation based on structured spatiotemporal sparsity as described in any of the schemes of the first aspect above is implemented.

[0028] In a fourth aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0029] The memory is used to store computer programs;

[0030] The processor is configured to implement the fast video generation method based on structured spatiotemporal sparsity as described in any one of the solutions of the first aspect when executing the computer program.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] Through an in-depth analysis of the Video Diffusion Transformers model, this paper reveals the existence of stable and hierarchically structured sparse patterns in its attention mechanism. Specifically, different attention heads spontaneously develop functional specializations, exhibiting different spatiotemporal attention behaviors, such as focusing on the local spatial region around the target location (local pattern); forming attention corridors running through the horizontal or vertical axis (cross pattern); or maintaining attention across the entire spatial image (global pattern). These patterns remain highly stable across different input and model levels, making offline precomputation possible.

[0033] This paper designs an automated offline mask search method. This algorithm formalizes the mask optimization problem as a hierarchical boundary contraction process. Starting from a complete attention window, the algorithm iteratively contracts the window boundary based on the ratio of recall loss to computational cost, prioritizing key attention regions. This process is controlled by both recall and cost thresholds to achieve a balance between model performance and computational speedup.

[0034] Ultimately, the proposed method significantly improves video generation speed and reduces computing resource requirements, with minimal or no sacrifice in video quality. Experimental results on both pure vision tasks and video generation tasks demonstrate that the proposed method significantly improves the efficiency and performance of the model in processing spatiotemporal information. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of the overall process of the method of the present invention;

[0036] Figure 2 Schematic diagram of the attention graph sparse pattern identified by the method of the present invention;

[0037] Figure 3 Schematic diagram showing the characteristics of different attention modes in the method of the present invention;

[0038] Figure 4 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0040] This paper proposes a fast video generation method based on structured spatiotemporal sparsity. The core idea of ​​this method is to pre-calculate the optimal sparse attention masks that can represent the intrinsic attention patterns in the video generation model through offline analysis and search, and then directly apply these masks during online inference, thereby avoiding the runtime prediction overhead and maximizing the computational efficiency of the hardware.

[0041] like Figure 1As shown, in a preferred implementation of the present invention, the above-mentioned fast video generation method based on structured spatiotemporal sparsity includes the following steps S1 to S4. The specific implementation process is described in detail below.

[0042] S1. Offline analysis of the pre-trained video generation model to obtain the attention map generated by its internal self-attention module when processing video data.

[0043] In step S1 of this embodiment of the present invention, a pre-trained video generation model (such as Hunyuan or Wan2.1 based on the DiT architecture) is first analyzed offline. A series of representative text prompts are input to drive the video generation model to generate videos. During the video generation process, the complete attention maps of each attention head in all Transformer layers within the video generation model are captured and saved.

[0044] S2. Identify stable, structured spatiotemporal sparse patterns in the attention map, where the spatiotemporal sparse patterns include at least different patterns in the spatial dimension and the temporal dimension. The patterns in the spatial dimension include local patterns, cross patterns, and global patterns, and the patterns in the temporal dimension include time-varying patterns and time-invariant patterns.

[0045] In an embodiment of the present invention, by systematically analyzing these captured attention maps (e.g. Figure 2 As shown in Figure 2). This embodiment found that although the attention map looks complex as a whole, its sparsity is not completely random, but rather exhibits highly structured characteristics. Figure 3 As shown, several typical spatiotemporal attention patterns can be summarized:

[0046] 1. Patterns in spatial dimensions:

[0047] Local Patterns: Some attention heads focus on a compact neighborhood around the query token, which is crucial for synthesizing fine local details.

[0048] Cross-Shaped Patterns: Other attention heads exhibit direction sensitivity, with their attention weights extending along the horizontal and vertical axes to form a cross-shaped area of ​​attention.

[0049] Global Patterns: Some attention heads always maintain connectivity to the entire spatial picture regardless of the query token position, or their attention weights are concentrated on salient objects in the video.

[0050] 2. Patterns in the time dimension:

[0051] Time-Invariant Patterns: Attention distribution is independent of time, keeping consistent attention patterns across all time steps.

[0052] Time-Invariant Patterns: Attention distribution is independent of time, keeping consistent attention patterns across all time steps.

[0053] The stability of these patterns, i.e., they do not change dramatically with changes in input text prompts or random seeds, is the theoretical basis for the invention to adopt an offline search strategy.

[0054] S3, based on the identified spatio-temporal sparse patterns, automatically search for the optimal sparse attention mask configuration for each attention head of the video generation model from the offline mask search method, forming a sparse attention mask library.

[0055] It should be noted that the design of the sparse attention mask adopts a double attention window strategy, i.e., adaptively combining two or more complementary basic window shapes to approximate the cross-shaped pattern and local pattern, so that explicit pattern classification is not required during inference.

[0056] In step S3 of the embodiments of the invention, the specific process of searching for sparse attention mask configurations using the offline mask search method is as follows:

[0057] S31, in each attention head of the video generation model, divide the video data into multiple different frame groups, each frame group containing the same number of frames, and the number of frame groups for each attention head is the same.

[0058] In this embodiment, in order to handle temporal dynamics, first divide all frames of the video into multiple frame groups. Each frame group will perform mask search independently and have its own set of sparse attention mask configurations. For example, if the total video data contains 60 frames, it can be divided into 5 frame groups, each containing 12 frames. The frame group number starts from 0, frames 0-11 belong to group 0, frames 12-23 belong to group 1, frames 24-35 belong to group 2, and so on. It should be noted that in this invention, how to group (i.e., how many frames each group contains and how many frame groups in total) is summarized through continuous experiments, which can be understood as an empirical determination. Therefore, the grouping operation here can be selected by a person skilled in the art according to the actual situation, and the invention does not limit the grouping method.

[0059] S32. For an attention head, initialize a mask covering the entire area of ​​the attention map in each frame group, preset a minimum recall rate threshold and a maximum computational cost threshold, continuously iterate to shrink the mask boundary, and preferentially remove areas whose attention contribution does not reach the preset target in each iteration. When the attention recall rate of the mask is lower than the minimum recall rate threshold or the computational cost gain of further shrinkage is lower than the maximum computational cost threshold, the iteration terminates and outputs the sparse attention mask configuration of the attention head.

[0060] In this embodiment, during iterative shrinkage, the offline mask search method evaluates the "recall loss" (i.e., the loss of important attention weights) and "computational cost savings" associated with removing different regions (e.g., tiles). The algorithm prioritizes removing regions with the highest cost-effectiveness (i.e., regions with minimal recall loss but significant cost savings, i.e., regions that fall short of the target).

[0061] In this embodiment, the shrinking process is controlled by two hyperparameters: a minimum recall threshold τ and a maximum computational cost threshold λ. When further shrinking results in a recall below τ, or when the cost-effectiveness of shrinking (ΔRecall / ΔCost) exceeds λ, the search for the current frame grouping stops. This ensures a good balance between speedup and quality.

[0062] S33: Change the text prompt or random seed used, regenerate the sparse attention mask configuration of the attention head according to step S32, and then merge the sparse attention mask configurations of the attention head to obtain the optimal sparse attention mask configuration of the attention head;

[0063] In step S33 of this embodiment, the merging operation is a union operation to ensure that all attention regions identified as important under different search conditions are retained in the optimal sparse attention mask configuration.

[0064] S34. Process all attention heads of the video generation model according to steps S32 to S33, and form a final universal sparse attention mask library by configuring all optimal sparse attention masks.

[0065] In this embodiment, to obtain a universal sparse attention mask configuration that is robust to various inputs, the algorithm performs multiple searches using different text prompts and random seeds. These sparse attention mask configurations found with different input prompts and random seeds are then merged using a union method to form a final, universal sparse attention mask library. This conservative merging strategy ensures that all potential key attention regions are preserved.

[0066] Based on the above findings, we designed an automated offline mask search process, which aims to find an optimal sparse attention mask configuration for each attention head of the model.

[0067] S4. During the online inference phase of the video generation model, according to the currently processed attention head and the frame group to which it belongs, the corresponding pre-computed sparse attention mask is loaded from the sparse attention mask library and applied to the self-attention calculation to skip the calculation of non-critical areas, thereby achieving acceleration.

[0068] In this embodiment, the online reasoning stage of the video generation model is the actual video generation stage, and the method of the present invention no longer needs to perform any dynamic sparsity prediction. For each self-attention calculation in the video generation model, first, according to the layer in which it is located and the attention head, the window size of the corresponding indicator mask is directly loaded from the offline generated sparse attention mask library, and then the self-attention calculation is performed based on the hardware-aware tiled method. The query tensor Q, key tensor K and value tensor V are each divided into tiles in the sequence dimension. For a row vector of the query tensor tile, the column vector participating in the calculation in the key tensor is obtained through the sparse attention mask, and the row vector is dot-producted with the column vector participating in the calculation to calculate the attention score, so as to maintain the regularity of the calculation and improve the hardware efficiency.

[0069] Specifically, the sparse attention mask is an n×n matrix, where n represents the length of the video data. There are two values ​​0 and 1 in the matrix, 1 means participating in the calculation, and 0 means not participating in the calculation. The present invention is compatible with efficient attention implementations such as FlashAttention. After searching for the sparse attention mask, the Q, K, and V tensors are each divided into small blocks, and the searched sparse attention mask is applied to the matrix multiplication at the tile level, and attention calculations are performed at the tile level. For example, the mask of a row vector (Query) of the Q matrix is ​​[0,1,1,1,0]. For this row vector, the mask is 1 in the positions of columns 1, 2, and 3, indicating that it is necessary to calculate the dot product between the row vector and the Key vector corresponding to the 1st, 2nd, and 3rd columns of the K matrix. In other words, calculations are only performed between tiles in non-masked areas, thereby achieving significant computational acceleration while maintaining regularity in the calculation and facilitating GPU parallel processing.

[0070] It should be noted that in step S4 of the present invention, in order to utilize the temporal stability of the diffusion model (that is, the characteristic that the diffusion model behaves similarly in adjacent denoising steps), the above-mentioned video generation model adopts the DiT architecture, and reuses the same set of sparse attention masks in multiple consecutive denoising steps (the sparse attention masks used are updated only once every few steps) to reduce the frequency of mask loading and further reduce the overhead of mask loading.

[0071] The present invention will now use a specific example to demonstrate the application effect of the fast video generation method based on structured spatiotemporal sparsity described in S1 to S4 of the above embodiments in a specific application scenario, so as to facilitate understanding of the essence of the present invention.

[0072] Example

[0073] The specific implementation process of the structured spatiotemporal sparsity-based rapid video generation method used in this embodiment is as described above. The specific steps will not be elaborated on in detail. The following will only demonstrate its effectiveness using case data. This paper aims to verify the acceleration effect and quality preservation ability of the proposed structured spatiotemporal sparsity-based rapid video generation method in actual video generation tasks.

[0074] All experiments in this embodiment are performed on a single NVIDIA H800 GPU, using the industry-leading text-to-video generation model Hunyuan (version with a parameter scale of 127K) as the infrastructure.

[0075] Table 1. Performance and objective quality comparison of different methods on the Hunyuan model

[0076] method Sparsity SSIM PSNR MSE Latency(s) Speedup Full Attention 0％ - - - 1370.658 1.00x Sparse VideoGen 50.35％ 0.7254 20.4297 822.8567 1117.767 1.23x SpargeAttn 47.77％ 0.7794 23.5889 369.3112 1148.628 1.19x Method of the present invention 62.36％ 0.9040 30.0822 105.1957 546.504 2.51x Method of the present invention 52.90％ 0.9452 34.5506 35.1307 750.201 1.83x

[0077] In Table 1, each method generates a video with 129 frames and a resolution of 768*1280 on the Hunyuan model. The implementation of the comparison method used in this embodiment belongs to the prior art. These include: Full Attention, which comes from the prior art literature Dao T, Fu D, Ermon S, et al. FlashAttention: Fast and Memory-Efficient ExactAttention with IO-Awareness[J]. Advances in Neural Information ProcessingSystems. 2022: 16344-16359; Sparse VideoGen, which comes from the prior art literature Xi H, Yang S, Zhao Y, et al. Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity[J]. arXiv preprint arXiv: 2502.01776, 2025; SpargeAttn, which comes from the prior art literature Zhang J, Xiang C, Huang H, et al. Spargeattn: Accurate sparseattention accelerating any model inference[J]. arXiv preprint arXiv: 2502.18137, 2025.

[0078] In order to more comprehensively evaluate the visual effect of the generated video, this embodiment further uses the VBench benchmark for evaluation, and the results are shown in Table 2.

[0079] Table 2. Multi-dimensional quality evaluation of different methods on the VBench benchmark

[0080]

[0081]

[0082] As can be seen from Tables 1 and 2 above, our method achieves high objective quality while maintaining a high sparsity rate. It performs comparable to the full-attention baseline across several key perceptual quality metrics, including subject consistency, background consistency, aesthetic quality, and text-video alignment. This further demonstrates that while our method achieves computational acceleration, the generated videos maintain a high standard of human visual perception.

[0083] Through the above technical solution, the experimental data of this embodiment strongly proves that the fast video generation method based on structured spatiotemporal sparsity proposed in the present invention can accurately utilize the inherent structured spatiotemporal sparsity in the video generation model, while achieving more than 2.5 times of inference acceleration, while effectively maintaining high-quality video generation effects comparable to the full-attention baseline, solving the key bottleneck in the existing technology where speed and quality are difficult to balance, and has significant technological progress and broad application prospects.

[0084] In summary, this paper moves the complex dynamic sparse pattern discovery process offline and designs a mask search algorithm that preserves key information and adapts to spatiotemporal changes. Ultimately, this method achieves efficient, high-quality video generation during the inference phase. Experimental results show that this method achieves significant advantages in both speed and generation quality compared to full attention and other sparse attention methods.

[0085] It should also be noted that the fast video generation method based on structured spatiotemporal sparsity in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a fast video generation system based on structured spatiotemporal sparsity corresponding to the fast video generation method based on structured spatiotemporal sparsity provided in the above embodiment, such as Figure 4 As shown, it includes:

[0086] The attention map generation module is used to analyze the pre-trained video generation model offline and obtain the attention map generated by the internal self-attention module when processing video data;

[0087] a sparse pattern recognition module for identifying stable, structured spatiotemporal sparse patterns in the attention map, wherein the spatiotemporal sparse patterns include at least different patterns in the spatial dimension and the temporal dimension; wherein the patterns in the spatial dimension are local patterns, cross patterns, and global patterns, and the patterns in the temporal dimension are time-varying patterns and time-invariant patterns;

[0088] A mask library acquisition module is used to automatically search for the optimal sparse attention mask configuration for each attention head of the video generation model based on the identified spatiotemporal sparse patterns using an offline mask search method to form a sparse attention mask library;

[0089] The online inference module is used to load the corresponding pre-computed sparse attention mask from the sparse attention mask library according to the currently processed attention head and the frame group to which it belongs during the online inference stage of the video generation model, and apply it to the self-attention calculation to skip the calculation of non-critical areas, thereby achieving acceleration.

[0090] Similarly, based on the same inventive concept, another preferred embodiment of the present invention further provides a computer electronic device corresponding to the fast video generation method based on structured spatiotemporal sparsity provided in the above embodiment, which includes a memory and a processor;

[0091] The memory is used to store computer programs;

[0092] The processor is configured to implement the fast video generation method based on structured spatiotemporal sparsity in the above embodiment when executing the computer program.

[0093] Furthermore, the logic instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention.

[0094] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the fast video generation method based on structured spatiotemporal sparsity provided in the above embodiment, and a computer program is stored on the storage medium. When the computer program is executed by the processor, it can implement the fast video generation method based on structured spatiotemporal sparsity in the above embodiment.

[0095] It is understood that the storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage medium may be any medium capable of storing program code, such as a USB flash drive, a mobile hard drive, a magnetic disk, or an optical disk.

[0096] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0097] It should also be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division. In actual implementation, there may be other division methods, for example, multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0098] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A fast video generation method based on structured spatiotemporal sparsity, characterized by: The following steps are involved: S1. Offline analysis of the pre-trained video generation model to obtain the attention map generated by its internal self-attention module when processing video data; S2. Identify stable, structured spatiotemporal sparse patterns in the attention map, where the spatiotemporal sparse patterns include at least different patterns in spatial and temporal dimensions; wherein the patterns in spatial dimensions are local patterns, cross patterns, and global patterns, and the patterns in temporal dimensions are time-varying patterns and time-invariant patterns; S3. Based on the identified spatiotemporal sparse patterns, an offline mask search method automatically searches for the optimal sparse attention mask configuration for each attention head of the video generation model to form a sparse attention mask library; S4. During the online inference phase of the video generation model, according to the currently processed attention head and the frame group to which it belongs, the corresponding pre-computed sparse attention mask is loaded from the sparse attention mask library and applied to the self-attention calculation to skip the calculation of non-critical areas, thereby achieving acceleration.

2. The method for fast video generation based on structured spatiotemporal sparsity according to claim 1, wherein: In step S1, a pre-trained video generation model is first analyzed offline. By inputting a series of representative text prompts, the video generation model is driven to generate videos. During the video generation process, the complete attention map of each attention head in all Transformer layers within the video generation model is captured and saved.

3. The method for fast video generation based on structured spatiotemporal sparsity according to claim 2, wherein: In step S3, the design of the sparse attention mask adopts a dual attention window strategy.

4. The method for fast video generation based on structured spatiotemporal sparsity according to claim 3, wherein: In step S3, the specific process of searching for the sparse attention mask configuration using the offline mask search method is as follows: S31. In each attention head of the video generation model, divide the video data into multiple different frame groups, each frame group contains the same number of frames, and the number of frame groups in each attention head is the same; S32. For an attention head, initialize a mask covering the area where the entire attention map is located in each frame group, preset a minimum recall rate threshold and a maximum computational cost threshold, continuously iterate to shrink the mask boundary, and preferentially remove areas whose attention contribution does not reach the preset target in each iteration. When the attention recall rate of the mask is lower than the minimum recall rate threshold or the computational cost gain of further shrinkage is lower than the maximum computational cost threshold, the iteration terminates and outputs the sparse attention mask configuration of the attention head; S33: Change the text prompt or random seed used, regenerate the sparse attention mask configuration of the attention head according to step S32, and then merge the sparse attention mask configurations of the attention head to obtain the optimal sparse attention mask configuration of the attention head; S34. Process all attention heads of the video generation model according to steps S32 to S33, and form a final universal sparse attention mask library by configuring all optimal sparse attention masks.

5. The method for fast video generation based on structured spatiotemporal sparsity according to claim 4, wherein: In step S33, the merging operation is a union operation.

6. The method for fast video generation based on structured spatiotemporal sparsity according to claim 1, wherein: In step S4, for each self-attention calculation in the video generation model, first, according to the layer and attention head in which it is located, the window size of the corresponding indication mask is directly loaded from the offline generated sparse attention mask library, and then the self-attention calculation is performed based on the hardware-aware tile method, and the query tensor, key tensor and value tensor are each divided into tiles in the sequence dimension. For a row vector of the query tensor tile, the column vector participating in the calculation in the key tensor is obtained through the sparse attention mask, and the row vector is dot-producted with the column vector participating in the calculation to calculate the attention score, so as to maintain the regularity of the calculation and improve hardware efficiency.

7. The method for fast video generation based on structured spatiotemporal sparsity according to claim 6, wherein: In step S4, the video generation model adopts the DiT architecture and reuses the same set of sparse attention masks in multiple consecutive denoising steps to reduce the frequency of mask loading.

8. A fast video generation system based on structured spatiotemporal sparsity, characterized by: include: The attention map generation module is used to analyze the pre-trained video generation model offline and obtain the attention map generated by the internal self-attention module when processing video data; a sparse pattern recognition module for identifying stable, structured spatiotemporal sparse patterns in the attention map, wherein the spatiotemporal sparse patterns include at least different patterns in the spatial dimension and the temporal dimension; wherein the patterns in the spatial dimension are local patterns, cross patterns, and global patterns, and the patterns in the temporal dimension are time-varying patterns and time-invariant patterns; A mask library acquisition module is used to automatically search for the optimal sparse attention mask configuration for each attention head of the video generation model based on the identified spatiotemporal sparse patterns using an offline mask search method to form a sparse attention mask library; The online inference module is used to load the corresponding pre-computed sparse attention mask from the sparse attention mask library according to the currently processed attention head and the frame group to which it belongs during the online inference stage of the video generation model, and apply it to the self-attention calculation to skip the calculation of non-critical areas, thereby achieving acceleration.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method for fast video generation based on structured spatiotemporal sparsity according to any one of claims 1 to 7 is implemented.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the fast video generation method based on structured spatiotemporal sparsity according to any one of claims 1 to 7 when executing the computer program.

Images