A method and system for generating streaming animation videos based on dynamic sparse attention.

Abstract

This invention discloses a method and system for generating streaming animation videos based on dynamic sparse attention. The method includes: text encoding, video latent vector initialization, spatiotemporal joint denoising, streaming feature continuation, periodic re-anchoring, and video decoding. The system includes a text encoding module, an AnimeStreamDiT model module, a two-branch streaming context encoding module, a periodic re-anchoring module, a video decoding module, and a training optimization module. This invention reduces memory consumption through temporal dynamic sparse attention, maintains temporal continuity through two-branch context encoding, and suppresses long-term degradation through periodic re-anchoring, achieving stable, efficient, and unlimited-duration animation video generation, suitable for animation creation, intelligent content generation, and other scenarios.

Description

[0001] Technical Field This invention relates to the fields of artificial intelligence, computer vision, and video generation technology, specifically to a method and system for generating streaming animation videos based on dynamic sparse attention.

[0002] Background Technology: Current text-driven video generation technology is widely used in animation creation, short video production, and film and television production. However, existing technologies suffer from problems such as high memory consumption, unstable temporal sequences in long videos, susceptibility to character breakdowns in animation, and lack of context in streaming generation. Traditional full-scale spatiotemporal attention computation complexity increases quadratically with the number of frames, making it unsuitable for generating long sequences. Streaming generation methods often employ simple splicing or mean aggregation, which easily leads to image fragmentation, content disjointness, and quality degradation. General video generation models lack animation style constraints, making it difficult to maintain consistency in character facial features, lines, and movements. These problems prevent existing technologies from achieving stable, efficient, and long-duration animation video generation, limiting their engineering implementation and commercial applications.

[0003] To address the shortcomings of existing technologies, this invention proposes a method and system for generating streaming animation videos based on dynamic sparse attention. By reducing memory consumption through temporal dynamic sparse attention, maintaining temporal coherence through a dual-branch streaming context encoder, and suppressing error accumulation through periodic re-anchoring, this invention achieves the generation of animation videos with unlimited duration, high stability, and high consistency.

[0004] The present invention adopts the following technical solution: a method for generating streaming animation videos based on dynamic sparse attention, including text encoding, latent vector initialization, spatiotemporal joint noise reduction, streaming feature continuation, periodic re-anchoring and video decoding steps.

[0005] An animation streaming video generation system based on dynamic sparse attention includes a text encoding module, a latent vector initialization module, an AnimeStreamDiT model module, a two-branch streaming context encoding module, a periodic re-anchoring module, a video decoding module, and a training optimization module.

[0006] The beneficial effects of this invention are: it significantly reduces memory usage through temporal dynamic sparse attention; it improves long-term temporal coherence through dual-branch context coding; it achieves unlimited duration generation without degradation through periodic re-anchoring; and it maintains character and style stability through animation motion anchors, thus possessing outstanding engineering practicality and commercial value.

[0007] Figure 1 is a diagram of the overall system architecture of the present invention; Figure 2 is a diagram of the AnimeStreamDiT model structure of the present invention; Figure 3 is a diagram of the working principle of the temporal dynamic sparse attention of the present invention; Figure 4 is a diagram of the structure of the dual-branch streaming context encoder of the present invention; Figure 5 is a flowchart of the periodic re-anchoring mechanism of the present invention; Figure 6 is a flowchart of the video generation method steps of the present invention.

[0008] The present invention will be further described in detail below with reference to specific embodiments.

[0009] Example 1: A method for generating streaming animation videos based on dynamic sparse attention, comprising the following steps: 1. Obtain text prompts and obtain text feature vectors using a CLIP text encoder. 2. Initialize a five-dimensional video latent space vector with a shape of (1, 4, 32, 64, 64). 3. Input the latent vector and text features into the AnimeStreamDiT model. High-motion regions are filtered for sparse computation using temporal dynamic sparse attention, spatial structure is extracted using spatial attention, and semantic alignment is achieved through cross-attention. 4. Historical features and current frame features are modeled using LSTM and window attention using a dual-branch streaming context encoder. Continuous features are obtained through gating. 5. Re-anchoring is performed every 64 frames, resetting the anchor features using the current high-confidence frame and normalizing them. 6. The latent features are decoded into continuous animation videos using a VAE decoder.

[0010] Example 2 presents an animation streaming video generation system based on dynamic sparse attention, comprising a text encoding module, a latent vector initialization module, an AnimeStreamDiT model module, a two-branch streaming context encoding module, a periodic re-anchoring module, a video decoding module, and a training optimization module. These modules work collaboratively to achieve text-driven generation of animation videos with unlimited duration.

[0011] Example 3: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in Example 1.

Claims

1. An animation streaming video generation method based on dynamic sparse attention, characterized in that, Includes the following steps: Text encoding steps: Obtain text prompt information, encode it using a text encoder to obtain text feature vectors; latent... Vector initialization step: Initialize a five-dimensional video latent space vector containing temporal and spatial dimensions; Spatiotemporal joint denoising step: Input the video latent space vector and text feature vector into the AnimeStreamDiT model, perform sparse attention calculation on high-motion regions through the temporal dynamic sparse attention module, extract single-frame spatial structure features through the spatial attention module, and achieve semantic alignment between text features and video features through the cross-attention module to complete the denoising process; Streaming feature continuation step: Perform LSTM short-term motion modeling and window attention long-range association modeling on historical video features and current frame features through a dual-branch streaming context encoder, and fuse them through a gating mechanism to obtain streaming continuation features; Periodic re-anchoring steps: Reset anchor point features at fixed frame intervals, normalize and constrain the latent vectors, and cut off the error accumulation path; Video decoding steps: Input the latent features of the denoised video into the video decoder to generate a continuous and time-consistent animation video.

2. The method according to claim 1, characterized in that, The temporal dynamic sparse attention module selects 20%-30% of high-motion frames based on the motion intensity output by the motion prediction network for attention calculation, while keeping the original features of the remaining frames unchanged.

3. The method according to claim 1, characterized in that, The dual-branch streaming context encoder includes an LSTM branch, a window attention branch, and a gating fusion layer, which can simultaneously preserve local motion information and global context information.

4. The method according to claim 1, characterized in that, The periodic re-anchoring step is performed every 64 frames, regenerating anchor features using the latent features of the current high-confidence frame, and normalizing the features using the L2 norm.

5. A streaming animation video generation system based on dynamic sparse attention, characterized in that, include: The text encoding module is used to convert text prompts into text feature vectors; potential The vector initialization module is used to generate five-dimensional video latent space vectors; The AnimeStreamDiT model module is used to implement spatiotemporal joint denoising, including temporal dynamic sparse attention, spatial attention, and cross attention; the dual-branch streaming context encoding module is used to fuse historical features and current frame features to generate continuous features; The periodic re-anchoring module is used to periodically reset anchor point features and suppress feature drift; the video decoding module is used to decode latent features into animation videos; and the training optimization module is used to implement mixed precision training, gradient checkpointing, and multi-GPU distributed training.

6. The system according to claim 5, characterized in that, The system can generate animated videos of 16, 32, 64 frames or more with no time limit, while maintaining consistency in characters, lines and style.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 4.

Images