Autoregressive video diffusion model key-value cache compression method and system

Abstract

This application relates to a key-value cache compression method and system for an autoregressive video diffusion model. The method includes: dividing the attention heads in the autoregressive video diffusion model to determine at least static attention heads and dynamic attention heads; performing structured pruning on the first key-value cache corresponding to the static attention head to obtain a static head compressed cache; dividing each frame of the historical video segment in the second key-value cache corresponding to the dynamic attention head into several image blocks, and retaining the key-value cache corresponding to the image blocks that meet preset conditions based on the degree of information change of the same image block between adjacent frames to obtain a dynamic head compressed cache; and generating the current video segment based on the static head compressed cache and the dynamic head compressed cache. This application solves the technical problem in related technologies where a unified key-value cache compression strategy cannot take into account the differentiated needs of different attention heads for historical context, leading to a decrease in the quality of the generated video.

Description

Technical Field

[0001] This application relates to the field of video generation technology, and in particular to a method and system for key-value caching compression of an autoregressive video diffusion model. Background Technology

[0002] Autoregressive video diffusion models generate complete videos by dividing them into consecutive video segments, thus supporting long-duration, interactive video generation tasks. To maintain temporal consistency and visual continuity across video segments, these models need to continuously store key-value caches corresponding to historical video segments during inference and use attention mechanisms to allow newly generated video segments to access the context of historical video segments. This leads to a sharp increase in the memory usage of the key-value cache and the computational overhead of attention as the generated video duration, resolution, and attention window expand, severely restricting the real-time deployment and practical application of these models. Therefore, how to efficiently compress the key-value cache while maintaining video generation quality is a pressing problem in this field when using autoregressive video diffusion models.

[0003] Existing techniques typically employ a uniform key-value caching compression strategy for all attention heads in an autoregressive video diffusion model during the inference phase. This could involve using a sliding window attention mechanism to retain only the key-value cache of the most recent frames, or performing uniform pruning on all historical frames based on a global importance score. The commonality among these existing solutions is that performing the same key-value caching compression operation indiscriminately on all attention heads can reduce the size of the key-value cache to some extent. However, in autoregressive video diffusion models, different attention heads differ in visual function. Therefore, performing indiscriminate key-value caching compression on all attention heads makes it difficult to consider the differentiated needs of different attention heads for historical context, leading to a decrease in the quality of the generated video.

[0004] There is currently no effective solution to the problem that the unified key-value caching compression strategy used in related technologies for autoregressive video diffusion models cannot take into account the different needs of different attention heads for historical context, resulting in a decline in the quality of generated videos. Summary of the Invention

[0005] This application provides a method and system for key-value caching compression of autoregressive video diffusion models to solve the technical problem that related technologies use a uniform key-value caching compression strategy for autoregressive video diffusion models, which cannot take into account the differentiated needs of different attention heads for historical context, resulting in a decrease in the quality of generated videos.

[0006] In a first aspect, this application provides an autoregressive video diffusion model key-value caching compression method, comprising:

[0007] The attention heads in the autoregressive video diffusion model are divided, at least identifying static attention heads and dynamic attention heads;

[0008] Structural pruning is performed on the first key-value cache corresponding to the static attention header to obtain a compressed static header cache;

[0009] Each frame of the historical video segment in the second key-value cache corresponding to the dynamic attention head is divided into several image blocks; based on the degree of information change of the same image block between adjacent frames, the key-value cache corresponding to the image block that meets the preset conditions is retained to obtain the dynamic head compression cache.

[0010] The current video segment is generated based on the static header compression cache and the dynamic header compression cache.

[0011] In some embodiments, the partitioning of attention heads in the autoregressive video diffusion model, at least determining static attention heads and dynamic attention heads, includes:

[0012] Before performing inference using the autoregressive video diffusion model, the attention distribution of each attention head in the autoregressive video diffusion model is obtained;

[0013] Based on the attention distribution, the corresponding attention heads are divided into static attention heads or dynamic attention heads.

[0014] In some further embodiments, obtaining the attention distribution of each attention head in the autoregressive video diffusion model includes:

[0015] The attention quality of each attention head assigned to different frame regions along the key dimension is statistically analyzed.

[0016] Calculate the proportion of attention distribution for each attention head in each frame region, and use it as the attention distribution.

[0017] In some further embodiments, the frame region includes at least: the current video segment, the transition anchor frame, and the full attention window;

[0018] The step of classifying the corresponding attention heads into static attention heads or dynamic attention heads based on the attention distribution includes:

[0019] When the attention head corresponding to the attention distribution ratio indicator focuses more than a preset percentage of attention on the current video segment and the transition anchor frame, the attention head is determined to be the static attention head;

[0020] When the attention distribution ratio indicates that the attention of the attention head is evenly distributed on the complete attention window, the attention head is determined to be the dynamic attention head.

[0021] In some embodiments, performing structured pruning on the first key-value cache corresponding to the static attention head to obtain a compressed static header cache includes:

[0022] In the first key-value cache, retrieve the key-value cache corresponding to the current video segment;

[0023] The key-value cache corresponding to the sink frame and the key-value cache corresponding to the transition anchor frame in the historical video segment adjacent to the current video segment are determined as the first historical key-value cache.

[0024] In the key-value caches corresponding to all historical video segments, pruning is performed on the key-value caches other than the first historical key-value cache.

[0025] In some embodiments, the step of retaining key-value caches corresponding to image blocks that meet preset conditions based on the degree of information change of the same image block between adjacent frames to obtain a dynamic header compression cache includes:

[0026] Calculate the key state similarity of the same image patch between adjacent frames;

[0027] The degree of information change is determined based on the key state similarity. When the degree of information change meets a preset condition, the corresponding image block is determined as an image block to be retained, and the key value cache corresponding to the image block to be retained is retained.

[0028] In some further embodiments, calculating the key state similarity of the same image patch between adjacent frames includes:

[0029] Obtain the key states of the first layer of the diffusion Transformer in the autoregressive video diffusion model;

[0030] Based on the key states of the first layer of the diffusion Transformer, the cosine similarity of the same image patch between adjacent frames is calculated as the key state similarity.

[0031] In some further embodiments, determining the degree of information change based on the key state similarity includes:

[0032] The key state similarity is negatively correlated with the degree of information change of the corresponding image block.

[0033] In some further embodiments, the step of determining the corresponding image block as an image block to be retained when the degree of information change meets a preset condition includes:

[0034] The image blocks whose key state similarity is lower than a preset threshold are identified as the image blocks to be retained.

[0035] Alternatively, the image blocks can be sorted from low to high according to the key state similarity, and a preset number of the image blocks can be selected from low to high to determine the image blocks to be retained.

[0036] In some embodiments, generating the current video segment based on the static header compression cache and the dynamic header compression cache includes:

[0037] All the static header compression caches are merged with all the dynamic header compression caches to obtain a compressed key-value cache;

[0038] When the autoregressive video diffusion model performs inference, it performs attention calculation on the compressed key-value cache using the query vector of the current video segment to generate the current video segment.

[0039] Secondly, this application provides an autoregressive video diffusion model key-value caching compression system, comprising:

[0040] The partitioning module is used to partition the attention heads in the autoregressive video diffusion model, at least determining the static attention head and the dynamic attention head;

[0041] The first compression module is used to perform structured pruning on the first key-value cache corresponding to the static attention head to obtain a static head compressed cache;

[0042] The second compression module is used to divide each frame of the historical video segment in the second key-value cache corresponding to the dynamic attention head into several image blocks, and retain the key-value cache corresponding to the image block that meets the preset conditions according to the degree of information change of the same image block between adjacent frames, so as to obtain the dynamic head compression cache.

[0043] The generation module is used to generate the current video segment based on the static header compression cache and the dynamic header compression cache.

[0044] Thirdly, in this application embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the autoregressive video diffusion model key-value caching compression method described in the first aspect above.

[0045] Fourthly, in this application embodiment, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the autoregressive video diffusion model key-value caching compression method described in the first aspect above.

[0046] Compared with existing related technologies, the embodiments of this application have the following beneficial effects:

[0047] This application divides the attention heads in the autoregressive video diffusion model into static attention heads and dynamic attention heads, and applies different key-value caching compression strategies to the two types of attention heads. For static attention heads, key-value caching related to the continuity of segment boundaries is preserved through structured pruning; for dynamic attention heads, historical video segments are divided into image blocks, and key-value caching is selectively preserved based on the degree of information change of the same image block between adjacent frames. Thus, while reducing the size of the key-value caching, lowering memory usage and attention computation overhead, this application can also accommodate the differentiated needs of different attention heads for historical context, avoiding discontinuous segment boundary transitions and dynamic quality degradation caused by a uniform compression strategy. Therefore, this application can efficiently compress the key-value caching while maintaining the quality of video generation.

[0048] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0049] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0050] Figure 1 This is a hardware structure block diagram of the terminal of the autoregressive video diffusion model key-value caching compression method provided in an embodiment of this application;

[0051] Figure 2 This is a flowchart of an autoregressive video diffusion model key-value caching compression method provided in an embodiment of this application;

[0052] Figure 3 This is a flowchart of obtaining a static header compression cache provided in one embodiment of this application;

[0053] Figure 4 This is a flowchart of obtaining a dynamic header compression cache according to an embodiment of this application;

[0054] Figure 5 This is a flowchart of determining the image block to be retained according to an embodiment of this application;

[0055] Figure 6This is a flowchart of generating the current video segment based on a static header compression buffer and a dynamic header compression buffer, provided in one embodiment of this application;

[0056] Figure 7 This is a schematic diagram illustrating the principle of key-value cache compression provided in an embodiment of this application;

[0057] Figure 8 This is a visual comparison diagram of aerial panoramic videos generated by different models according to an embodiment of this application;

[0058] Figure 9 This is a visual comparison diagram of different models generating videos of people walking, provided in an embodiment of this application.

[0059] In the diagram: 102, processor; 104, memory; 106, transmission device; 108, input / output device. Detailed Implementation

[0060] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0061] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order.

[0062] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1This is a hardware structure block diagram of the terminal for the autoregressive video diffusion model key-value caching compression method in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0063] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the autoregressive video diffusion model key-value caching compression method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0064] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0065] This embodiment provides a key-value caching compression method for autoregressive video diffusion models, applicable to inference acceleration and memory optimization in long-duration, high-resolution video generation scenarios. This type of model generates videos by dividing them into continuous video segments for streaming. During inference, it needs to continuously store the key-value cache corresponding to historical video segments, leading to a sharp increase in memory usage and attention computation overhead with the generation scale. This embodiment aims to achieve efficient key-value caching compression while maintaining the continuity of segment boundaries and dynamic quality of the generated video by applying differentiated key-value caching compression strategies to attention heads with different functions.

[0066] Figure 2 This is a flowchart of the autoregressive video diffusion model key-value caching compression method in this embodiment, as shown below. Figure 2 As shown, the process includes the following steps:

[0067] Step S210: Divide the attention heads in the autoregressive video diffusion model, and at least determine the static attention head and the dynamic attention head.

[0068] For example: a static attention head can refer to an attention head that focuses primarily on the current video segment and the most recent historical frames, used to maintain intra-frame details and boundary transitions of video segments; a dynamic attention head can refer to an attention head whose attention is distributed across a wider range of historical frames beyond the most recent historical frame region, used to focus on the evolution of the same image patch across frames, used to maintain subject consistency.

[0069] Step S220: Perform structured pruning on the first key-value cache corresponding to the static attention header to obtain a compressed static header cache.

[0070] For example: Structured pruning can refer to the compression operation of selectively retaining and pruning the key-value cache corresponding to the static attention head based on the structural requirements of the continuity of video segment boundaries.

[0071] Step S230: Divide each frame of the historical video segment in the second key-value cache corresponding to the dynamic attention head into several image blocks; based on the degree of information change of the same image block between adjacent frames, retain the key-value cache corresponding to the image block that meets the preset conditions to obtain the dynamic head compression cache.

[0072] For example, the degree of information change can refer to the magnitude of content change of the same image patch between adjacent frames; for instance, image patches with a larger magnitude of change correspond to moving or evolving regions, while image patches with a smaller magnitude of change correspond to static or redundant regions. The magnitude mentioned here can be designed specifically for practical situations. This step reduces the size of the key-value cache while retaining cross-frame dynamic information by performing differential compression on the dynamic attention head based on the degree of information change of the image patch.

[0073] Step S240: Generate the current video segment based on the static header compression cache and the dynamic header compression cache.

[0074] In this step, the two types of compressed caches mentioned above are combined and used as the historical context accessed when generating the current video segment, so that the generation process can simultaneously obtain structural information related to boundary transitions and dynamic information related to motion evolution.

[0075] Existing techniques typically employ a uniform key-value caching compression strategy for all attention heads in autoregressive video diffusion models. For example, sliding window attention might retain only the key-value cache of the most recent frames, or all historical frames might be pruned uniformly based on a global importance score. This approach indiscriminately performs the same compression operation on all attention heads, ignoring the differences in visual function among them. This leads to discontinuous transitions at segment boundaries and degrades dynamic quality during motion.

[0076] Unlike existing technologies, this embodiment, through the aforementioned steps, first distinguishes the functional types of different attention heads, identifying their differences in local structure preservation and long-range motion modeling. Then, it applies differentiated compression strategies matching the functions of different attention head types. Specifically, for the key-value cache corresponding to static attention heads, this embodiment retains structural information related to the continuity of video segment boundaries. For the key-value cache corresponding to dynamic attention heads, this embodiment selectively retains information based on the degree of information change in different image blocks in historical frames, i.e., retaining areas with significant changes and removing static and redundant areas. This compresses the size of the key-value cache while preserving the key information required for cross-frame dynamics and subject evolution. Finally, this embodiment combines the compressed caches corresponding to each type to generate the current video segment.

[0077] It should be noted that the static attention head and dynamic attention head described in this embodiment are different from the static and dynamic concepts used in existing understandings to describe changes in image content. In this embodiment, the static attention head refers to an attention head that focuses on local details and transitions at video segment boundaries, while the dynamic attention head refers to an attention head that focuses on long-range motion consistency across frames and the evolution of the subject. Both are specific functional classifications defined based on the discovery of the inherent functional differentiation law of the attention mechanism in the autoregressive video diffusion model, rather than descriptions of static and moving areas in the video image.

[0078] Based on the aforementioned technical means, this embodiment can take into account the differentiated needs of different attention heads for historical context, thereby achieving efficient key-value caching compression while maintaining video generation quality. In view of this, this embodiment can solve the technical problem that related technologies, which employ a uniform key-value caching compression strategy for autoregressive video diffusion models, cannot take into account the differentiated needs of different attention heads for historical context, leading to a decrease in generated video quality.

[0079] In some embodiments, step S210, which involves dividing the attention heads in the autoregressive video diffusion model and determining at least the static attention heads and dynamic attention heads, specifically includes the following steps:

[0080] Step S211: Before performing inference using the autoregressive video diffusion model, obtain the attention distribution of each attention head in the autoregressive video diffusion model;

[0081] Step S212: Based on the attention distribution, the corresponding attention heads are divided into static attention heads or dynamic attention heads.

[0082] For example: Attention distribution can refer to the attention allocation of each attention head across frames of a historical video segment, used to characterize the differences in the degree of attention each attention head pays to different frame regions. The frame region mentioned here refers to a grouping of frames that the attention head focuses on in the time dimension, such as the frame region corresponding to the current video segment, the frame region corresponding to the most recent historical video segment, or the frame region corresponding to the complete attention window. The mentioned partitioning operation can refer to partitioning the attention head according to the attention concentration range indicated by the attention distribution.

[0083] This embodiment obtains the attention distribution before inference and explicitly uses the attention distribution as the basis for classifying the functional types of attention heads. It can identify the differences in visual functions of different attention heads, which facilitates the subsequent execution of differentiated compression strategies that match the functions of static and dynamic attention heads.

[0084] In some further embodiments, step S211, which is the step of obtaining the attention distribution of each attention head in the autoregressive video diffusion model, specifically includes the following steps:

[0085] Step S211.1: Calculate the attention quality of each attention head assigned to different frame regions along the key dimension;

[0086] Step S211.2: Calculate the attention distribution ratio of each attention head in each frame region, and use it as the attention distribution.

[0087] For example: "Key dimension" can refer to the dimension in which the key vector resides in the attention mechanism; "attention quality" can refer to the statistical quantity of the attention value assigned by the attention head to a specific frame region. As indicated in the preceding embodiments, "frame region" can refer to the grouping of frame regions along the time dimension. "Attention distribution ratio" can refer to the attention distribution of each attention head in different frame regions, obtained by first statistically analyzing the attention quality of each attention head assigned to different frame regions along the key dimension, and then calculating the ratio of each attention head in each frame region based on the statistical results.

[0088] This embodiment obtains the original attention allocation data of each attention head in different frame regions by statistically analyzing the attention quality of the key dimension, and then quantifies the attention distribution by calculating the attention distribution ratio in different frame regions.

[0089] In some feasible embodiments, when calculating the attention distribution ratio in step S211.2, each frame region includes the current video segment, the transition anchor frame, the sink frame, and the full attention window.

[0090] Specifically: the current video segment refers to the video segment being generated; transition anchor frames refer to the last few frames in adjacent historical video segments used to maintain the continuity of video segment boundaries; sink frames refer to specific frames in the model that have a fixed attention attraction effect; and the complete attention window refers to the attention range covering all historical frames. By including sink frames as independent frame regions in the calculation of attention distribution ratios, the fixed attention behavior of sink frames can be used to help determine the functional type of the attention head.

[0091] In some other feasible embodiments, when calculating the attention distribution ratio in step S211.2, each frame region includes the current video segment, the transition anchor frame, and the complete attention window. In this case, each frame region no longer treats the sink frame as an independent frame region and does not include the sink frame in the calculation of the attention distribution ratio, thereby avoiding interference from the fixed attention attraction of the sink frame in determining the function type of the attention head.

[0092] In some further embodiments, the frame region includes at least: the current video segment, the transition anchor frame, and the complete attention window; step S212, which is the step of dividing the corresponding attention head into a static attention head or a dynamic attention head according to the attention distribution, specifically includes the following steps:

[0093] Step S212.1: When the attention head corresponding to the attention distribution ratio indicator focuses more than a preset percentage of attention on the current video segment and the transition anchor frame, the attention head is determined to be a static attention head;

[0094] Step S212.2: When the attention of the attention head is evenly distributed across the complete attention window, as indicated by the attention distribution ratio, the attention head is determined to be a dynamic attention head.

[0095] For example: a preset percentage can refer to a pre-defined threshold for judging the degree of attention concentration, which can be set specifically in practical applications. The static attention head is primarily responsible for maintaining intra-frame details and the boundary continuity between adjacent video segments. Its attention is naturally focused on the currently generated content and adjacent transition areas. Therefore, the static attention head is classified based on the attention invested in the current video segment and the transition anchor frame. The dynamic attention head, on the other hand, is primarily responsible for tracking cross-frame dynamics and subject consistency. It needs to acquire evolutionary information of the same image patches within a wider range of historical frames. Therefore, its attention is evenly distributed across the entire attention window, and this is used as the basis for classifying dynamic attention heads.

[0096] This embodiment associates the functional differences of attention heads with their attention distribution characteristics, and uses preset percentages and uniform distribution as quantitative judgment conditions to quantitatively classify static attention heads and dynamic attention heads, thereby accurately classifying attention heads.

[0097] In some of these embodiments, Figure 3 This is a flowchart of the process for obtaining the static header compression cache provided in this embodiment. Please refer to it. Figure 3 Step S220, which involves performing structured pruning on the first key-value cache corresponding to the static attention head to obtain the static header compressed cache, specifically includes the following steps:

[0098] Step S221: In the first key-value cache, obtain the key-value cache corresponding to the current video segment, and determine the first historical key-value cache corresponding to the historical video segments that are continuously related to the boundary of the current video segment.

[0099] Step S222: In the key-value caches corresponding to all historical video segments, perform pruning on the key-value caches other than the first historical key-value cache.

[0100] For example: The first key-value cache, here referring to the initial key-value cache corresponding to the static attention head, is distinct from the second key-value cache corresponding to the dynamic attention head. The first historical key-value cache can refer to the key-value caches corresponding to historical video segments that are determined to be boundary-continuously related to the current video segment and therefore need to be retained within the first key-value cache. Boundary continuity can refer to the continuity of image content between adjacent video segments.

[0101] This embodiment obtains a static header compression cache by identifying and retaining the boundary-related pruning cache in the first historical key-value cache, while pruning the remaining historical key-value caches. This static header compression cache ensures that the key structural information required to maintain the continuity of video segment boundaries is maintained, without any abrupt changes in the video frame.

[0102] In some further embodiments, step S221, which involves determining the first historical key-value cache corresponding to historical video segments that are continuously related to the boundary of the current video segment, specifically includes the following steps:

[0103] The key-value cache corresponding to the sink frame, and the key-value cache corresponding to the transition anchor frame in the historical video segment adjacent to the current video segment, are determined as the first historical key-value cache.

[0104] In this embodiment, retaining the sink frame helps maintain the model's inherent attention structure, while retaining the transition anchor frame provides the contextual information needed for the connection between adjacent video segments. Retaining both ensures that the static attention head can maintain the boundary continuity of video segments even after compression.

[0105] In some feasible embodiments, step S211, obtaining the attention distribution of each attention head in the autoregressive video diffusion model, only needs to be performed once before performing inference using the autoregressive video diffusion model. This attention distribution remains stable across different cue word samples and different denoising steps, and can serve as the basis for distinguishing between static and dynamic attention heads in all subsequent inference steps.

[0106] In some of these embodiments, Figure 4 This is a flowchart of the process for obtaining the dynamic header compression cache provided in this embodiment. Please refer to it. Figure 4 In step S230, based on the degree of information change of the same image block between adjacent frames, the key-value cache corresponding to the image block that meets the preset conditions is retained to obtain the dynamic header compression cache. This sub-step specifically includes the following steps:

[0107] Step S231: Calculate the key state similarity of the same image patch between adjacent frames;

[0108] Step S232: Determine the degree of information change based on key state similarity. When the degree of information change meets the preset conditions, determine the corresponding image block as the image block to be retained and retain the key value cache corresponding to the image block to be retained.

[0109] For example: Key-state similarity refers to the degree of similarity of the key states of the same image patch between adjacent frames. Information change level refers to the magnitude of content change of the same image patch between adjacent frames. Image patches to be retained refer to image patches whose information change level meets preset conditions, and therefore their key-value caches need to be retained. In addition, the pruning operation is only performed on frames of historical video segments that have been generated. The key-value caches corresponding to the frames of the current video segment are all retained and do not participate in the evaluation of information change level or selective pruning.

[0110] This embodiment determines the degree of information change of the same image patch between adjacent frames by calculating key state similarity, and retains the key value cache corresponding to image patches whose information change degree meets preset conditions, so that the dynamic head compression cache focuses on the change information required for cross-frame dynamics and subject evolution. Thus, after removing the key value cache of static redundant regions, it is possible to retain dynamic information that is crucial to motion consistency while reducing the size of the key value cache.

[0111] In some further embodiments, step S231, which is the step of calculating the key state similarity of the same image patch between adjacent frames, specifically includes the following steps:

[0112] Step S231.1: Obtain the key states of the first layer of the diffusion Transformer in the autoregressive video diffusion model;

[0113] Step S231.2: Based on the key states of the first layer of the diffusion Transformer, calculate the cosine similarity of the same image patch between adjacent frames, and use it as the key state similarity.

[0114] For example: First-layer Diffusion Transformer can refer to the first-layer diffusion transformer network used for feature extraction in an autoregressive video diffusion model. Cosine similarity can refer to a measure of the degree of similarity between two vectors by calculating the cosine of the angle between them.

[0115] This embodiment obtains the key states of the first-layer diffusion Transformer and calculates the cosine similarity as the key state similarity. This allows the similarity calculation in the key-value cache compression process corresponding to the dynamic attention head to be performed only based on the first-layer diffusion Transformer, without repeating the calculation on all layers, thus reducing the additional computational overhead introduced by the compression process.

[0116] In some further embodiments, step S232, which involves determining the degree of information change based on key state similarity, specifically includes the following steps:

[0117] Key state similarity shows a negative correlation with the degree of information change in the corresponding image patch.

[0118] For example: negative correlation occurs, meaning that the lower the key state similarity, the greater the degree of information change in the corresponding image patch; conversely, the higher the key state similarity, the lower the degree of information change in the corresponding image patch. Specifically, low key state similarity usually indicates that the key state of the image patch has changed significantly between adjacent frames, and the corresponding image area may contain moving objects, pose changes, or scene evolution. High key state similarity usually indicates that the key state of the image patch remains stable between adjacent frames, and the corresponding image area may correspond to the background, stationary objects, or content with minimal changes.

[0119] This embodiment establishes a negative correlation between key state similarity and the degree of information change, transforming key state similarity into a quantifiable index of information change, thereby effectively distinguishing between moving / evolving regions and static / redundant regions based on similarity values.

[0120] In some further embodiments, Figure 5 This is a flowchart for determining the image blocks to be retained, provided in this embodiment. Please refer to it. Figure 5 In step S232, the sub-step of determining the corresponding image block as the image block to be retained when the degree of information change meets the preset conditions specifically includes the following steps:

[0121] Image blocks with a key state similarity lower than a preset threshold are identified as image blocks to be retained.

[0122] For example: a preset threshold can refer to a pre-set threshold for judging key state similarity. Image blocks that are below this threshold are considered to meet the condition for information change.

[0123] This embodiment uses a preset threshold method to determine the image blocks to be retained, which can filter out image blocks with obvious changes based on the absolute similarity standard.

[0124] In some further embodiments, step S232, which determines the corresponding image block as the image block to be retained when the degree of information change meets the preset conditions, may further include the following steps:

[0125] Sort the image blocks by button state similarity from low to high, and select a preset number of image blocks from low to high to determine the image blocks to be retained.

[0126] For example: preset quantity can refer to the number of image blocks to be retained in advance, which can be set specifically according to the target compression rate in actual applications.

[0127] This embodiment uses a sorting selection method to determine the image blocks to be retained, which can flexibly control the number of image blocks to be retained according to the target compression ratio.

[0128] In some of these embodiments, Figure 6 This is a flowchart provided in this embodiment for generating the current video segment based on the static header compression buffer and the dynamic header compression buffer. Please refer to it. Figure 6 Step S240 specifically includes the following steps:

[0129] Step S241: Merge all static header compression caches with all dynamic header compression caches to obtain a compressed key-value cache;

[0130] In step S242, when the autoregressive video diffusion model performs inference, attention calculation is performed on the compressed key-value cache using the query vector of the current video segment to generate the current video segment.

[0131] For example: The compressed key-value cache can refer to the complete cache obtained by merging the static header compressed cache and the dynamic header compressed cache, used as the historical context accessed when generating the current video segment. The query vector can refer to the query representation corresponding to the current video segment in the attention calculation, used for attention matching with the key vector in the compressed key-value cache.

[0132] This embodiment combines the static header compression buffer and the dynamic header compression buffer, enabling the generation of the current video segment to simultaneously acquire two types of information: structural information related to the continuity of video segment boundaries retained in the static header compression buffer, and dynamic information related to cross-frame motion evolution retained in the dynamic header compression buffer. Thus, while reducing the size of the key-value buffer, the continuity of segment boundaries and dynamic quality of the generated video segment can still be guaranteed.

[0133] In some feasible embodiments, Figure 7 A schematic diagram illustrating the principle of key-value caching compression. Figure 7 This mainly demonstrates the operation process of key-value cache retention and pruning based on the differences in attention head functions described in the aforementioned method embodiments and their feasible implementations.

[0134] Please refer to Figure 7 The left side: During the autoregressive video generation process, the model performs attention matching calculations by matching the query features of the current frame with the key features of historical frames. The distribution configuration of attention heads includes: for each attention head, its own attention distribution feature performance is evaluated, and a preset percentage or threshold is set as the classification criterion. When the attention distribution ratio of the attention head is concentrated on the current video segment and the transition anchor frame, exceeding the preset percentage or threshold, it is classified as a static attention head; when the attention is evenly distributed across the entire attention window, it is classified as a dynamic attention head.

[0135] Please refer to Figure 7The upper right static attention head branch: The attention matrix of the static attention head exhibits a clear characteristic of being concentrated along the diagonal and sink frames. Under this mechanism, for transition anchor frames at the boundaries of video segments, the model chooses to preserve their structural information as a source of context. The dark blue transition frames in the figure illustrate that the static attention head is responsible for maintaining the stability of the image structure during scene transitions, ensuring that visual elements do not collapse or flicker.

[0136] Please refer to Figure 7 The lower right branch of the dynamic attention head exhibits a non-diagonal, multi-frame distributed dependency distribution in its attention matrix. This branch introduces a quantization mechanism for image patch-level key-state similarity. The model evaluates the key-state similarity of the same image patch across different historical frames. Since key-state similarity is negatively correlated with the degree of information change, image patches with low similarity and high information change are retained; while redundant image patches with high similarity and low information change are removed through pruning.

[0137] In conclusion, Figure 7 The revealed differential compression strategy enables the model to retain the structural information required for video segment transitions through a static attention head, and to accurately extract the dynamic evolution information required for cross-frame motion through a dynamic attention head. This mechanism allows the autoregressive video diffusion model to effectively maintain the structural consistency and temporal coherence of video generation while improving inference speed and reducing the computational scale of key-value caching.

[0138] In some feasible embodiments, in order to fully verify the effect in the long video generation scenario, this embodiment conducted comparative experiments under different duration settings of the Long Live model and the Self Forcing model.

[0139] In the Long Live model's 60-second long video generation scenario, the frame rate with the full key-value cache setting is 20.48 frames per second. After adopting the steps demonstrated in the aforementioned method embodiments and feasible embodiments, the frame rate is increased to 26.71 frames per second, achieving a 1.30x inference speedup. Hereinafter, the aforementioned method embodiments and feasible embodiments will be referred to as this method. Regarding specific metrics, this method reduces the block drop metric from 2.6 to 2.5 and increases the dynamic range metric from 42.40 to 43.56. In general quality metrics, the total score of this method increases from 80.23 to 80.43, and maintains a level comparable to, or even better than, the full key-value cache setting in sub-metrics such as image quality, subject consistency, background consistency, motion smoothness, and aesthetic quality.

[0140] In the Long Live model's 30-second video generation scenario, the full key-value cache was set to 21.10 frames per second. Using this method, the frame rate increased to 26.77 frames per second, achieving a 1.27x inference speedup. The dynamic range score improved from 42.54 to 43.65, and the overall score increased from 80.38 to 80.65, while general quality metrics such as subject consistency and background consistency remained stable.

[0141] In the scenario of generating a 30-second long video using the Self Forcing model, the acceleration effect is more significant due to the relatively large computational load of the model. With the full key-value cache set at a frame rate of 17.76 frames per second, this method increases the frame rate to 26.65 frames per second, achieving a 1.50x inference speedup. Regarding specific metrics, the block drop metric significantly decreased from 3.4 to 2.7, and the dynamic range improved from 46.86 to 52.23. On general quality metrics, the total score increased from 79.72 to 80.07. Experimental data demonstrates that this method effectively avoids the dynamic quality degradation problem that easily occurs in existing unified compression methods while achieving a 1.50x inference speedup.

[0142] In short video generation scenarios, the method of this application also achieves a balance between efficiency and quality. In the 5-second short video generation scenario using the LongLive model, the frame rate with the full key-value cache set is 21.85 frames per second. After adopting the method of this application, the frame rate is increased to 29.58 frames per second, achieving a 1.35x inference speedup. The dynamic range is improved from 40.28 to 45.56, and the total score is improved from 83.19 to 83.23, with both image quality and semantic scores remaining at a high level.

[0143] In the scenario of generating a 5-second short video using the Self Forcing model, the frame rate with the full key-value cache was 19.56 frames per second. After adopting this method, the frame rate increased to 28.18 frames per second, achieving a 1.44x inference speedup. The dynamic range improved from 66.39 to 69.17, and the total score increased from 83.91 to 83.98. In general quality metrics such as image quality score and semantic score, this method also maintained performance comparable to or even better than the full key-value cache configuration.

[0144] Based on the experimental data from the various scenarios described above, the autoregressive video diffusion model key-value cache compression method in this embodiment does not simply compress the historical key-value cache uniformly. Instead, it retains the structural information of anchor frames required for the boundary transitions of video segments corresponding to static attention heads, and retains the dynamic information of image patches required for cross-frame motion evolution corresponding to dynamic attention heads, based on the differences in attention head functions. This hybrid compression strategy enables significant improvement in inference speed while maintaining or even surpassing the visual quality and temporal continuity under a complete key-value cache setting in long-duration, interactive, and high-resolution video generation scenarios.

[0145] In some feasible embodiments, after performing model key-value caching compression according to the steps described in the foregoing method embodiments and their feasible implementations, the effect of this method is compared with two existing methods in a typical long video generation scenario. Specifically, Figure 8 This is a visual comparison of aerial panoramic videos generated by different models, mainly showing the continuous frame results of 5-second aerial panoramic videos generated by the Long Live model, the Dummy-F model, and the Forcing-KV model of this method. Figure 9 This is a visual comparison of different models generating videos of a person walking. It mainly shows the continuous frame results of a 30-second video of a person walking generated by the Long Live model, the Dummy-F model, and the Forcing-KV model of our method. Both images show continuous frames progressing over time from left to right. Figure 8 and Figure 9 In the text, the first existing method refers to generating videos using the Long Live model, and the second existing method refers to generating videos using the Dummy-F model.

[0146] Please refer to Figure 8 The first line shows the continuous frame results of the video generated using the Long Live model; the second line shows the continuous frame results of the video generated using the Dummy-F model, which uses uniform compression for the key-value cache of all attention heads; the third line shows the continuous frame results of the video generated using the Forcing-KV model of this method. In the 5-second aerial panoramic video generation, the corresponding input prompt is "Aerial panoramic video taken by a drone flying over a fantastic landscape." Observation Figure 8 It can be seen that the LongLive model, as the baseline generation model, can generate fantasy landscapes with complete structure. The Dummy-F model is prone to local image distortion and inter-frame flickering in the same scene. The Forcing-KV model of this method can maintain the structural stability of image elements such as sky, mountains and lakes, without abrupt geometric distortion and image flickering.

[0147] Please refer to Figure 9Similarly, the first row shows the continuous frame results of the video generated using the Long Live model; the second row shows the continuous frame results of the video generated using the Dummy-F model, which uses uniform compression for the key-value cache of all attention heads; the third row shows the continuous frame results of the video generated using the Forcing-KV model of this method. In the generation of a 30-second walking video, the corresponding input prompt is "a close-up of a man made entirely of rock, walking through a dense forest." Observation Figure 9 It can be seen that the Long Live model, as the baseline generation model, can generate continuous walking scenes. The Dummy-F model is prone to problems such as target loss and scene degradation during long-term generation. The Forcing-KV model of our method stably maintains the integrity of the main features of the character during long-term generation, and the background, lighting and shadow and character pose are consistent and coherent.

[0148] In some feasible embodiments, after performing model key-value caching compression according to the steps described in the foregoing method embodiments and their feasible implementations, the long video generation effect of the Forcing-KV model of this method is compared with that of the LongLive model and the Dummy-F model in an interactive long video generation scenario.

[0149] In a 60-second interactive video generation scenario, comparative experiments revealed that while the Long Live model can generate complete narrative footage, the transitions between actions appear somewhat stiff during continuous generation over 60 seconds. The Dummy-F model, when handling such continuous shots, is prone to issues such as inconsistent character orientation, structural inconsistencies, or inter-frame flickering. Our proposed Forcing-KV model, however, maintains a stable frame rate while ensuring the coherent execution of complex action sequences. Facial features and clothing details remain highly consistent across the 60 seconds, and background flickering and geometric distortion are significantly reduced, effectively avoiding temporal degradation.

[0150] This application provides an autoregressive video diffusion model key-value caching compression system, comprising:

[0151] The partitioning module is used to partition the attention heads in the autoregressive video diffusion model, at least determining the static attention head and the dynamic attention head;

[0152] The first compression module is used to perform structured pruning on the first key-value cache corresponding to the static attention header to obtain a compressed static header cache.

[0153] The second compression module is used to divide each frame of the historical video segment in the second key-value cache corresponding to the dynamic attention head into several image blocks. Based on the degree of information change of the same image block between adjacent frames, the key-value cache corresponding to the image block that meets the preset conditions is retained to obtain the dynamic head compression cache.

[0154] The generation module is used to generate the current video segment based on the static header compression cache and the dynamic header compression cache.

[0155] In some embodiments, the partitioning module is further configured to obtain the attention distribution of each attention head in the autoregressive video diffusion model before performing inference using the autoregressive video diffusion model; and to partition the corresponding attention heads into static attention heads or dynamic attention heads according to the attention distribution.

[0156] In some further embodiments, the partitioning module is also used to statistically analyze the attention quality of each attention head allocated to different frame regions along the key dimension; and to calculate the attention distribution ratio of each attention head in each frame region as the attention distribution.

[0157] In some further embodiments, the frame region includes at least: the current video segment, the transition anchor frame, and the complete attention window; the segmentation module is further configured to determine the attention head as a static attention head when the attention distribution ratio indicates that the attention head is concentrated on the current video segment and the transition anchor frame by more than a preset percentage; and to determine the attention head as a dynamic attention head when the attention distribution ratio indicates that the attention head is evenly distributed on the complete attention window.

[0158] In some embodiments, the first compression module is further configured to: obtain the key value cache corresponding to the current video segment in the first key value cache; determine the first historical key value cache corresponding to the historical video segments that are continuously related to the boundary of the current video segment; and perform pruning on the key value caches other than the first historical key value cache in all the key value caches corresponding to the historical video segments.

[0159] In some further embodiments, the first compression module is also used to determine the key value cache corresponding to the sink frame and the key value cache corresponding to the transition anchor frame in the historical video segment adjacent to the current video segment as the first historical key value cache.

[0160] In some embodiments, the second compression module is further configured to calculate the key state similarity of the same image block between adjacent frames; determine the degree of information change based on the key state similarity; and when the degree of information change meets a preset condition, determine the corresponding image block as an image block to be retained and retain the key value cache corresponding to the image block to be retained.

[0161] In some further embodiments, the second compression module is also used to obtain the key state of the first layer diffusion Transformer in the autoregressive video diffusion model; based on the key state of the first layer diffusion Transformer, to calculate the cosine similarity of the same image patch between adjacent frames as the key state similarity.

[0162] In some further embodiments, the second compression module is also used to determine the degree of information change based on the negative correlation between the key state similarity and the degree of information change of the corresponding image block.

[0163] In some further embodiments, the second compression module is also used to determine image blocks with key state similarity lower than a preset threshold as image blocks to be retained; or, sort the image blocks by key state similarity from low to high, select a preset number of image blocks from low to high, and determine them as image blocks to be retained.

[0164] In some embodiments, the generation module is further configured to merge all static header compression caches with all dynamic header compression caches to obtain a compressed key-value cache; and to perform attention calculation on the compressed key-value cache using the query vector of the current video segment during inference in the autoregressive video diffusion model to generate the current video segment.

[0165] The terms "module," "unit," "subunit," etc., used above can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the above embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0166] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0167] This embodiment also provides a computer device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0168] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0169] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0170] Furthermore, in conjunction with the autoregressive video diffusion model key-value caching compression method provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any one of the autoregressive video diffusion model key-value caching compression methods in the above embodiments.

[0171] It should be noted that all information and data involved in this application are authorized by the user or fully authorized by all parties and will be used legally.

[0172] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0173] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0174] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0175] The above embodiments merely illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application.

Claims

1. An autoregressive video diffusion model key-value cache compression method, characterized in that, include: The attention heads in the autoregressive video diffusion model are divided, at least identifying static attention heads and dynamic attention heads; Structural pruning is performed on the first key-value cache corresponding to the static attention header to obtain a compressed static header cache; Each frame of the historical video segment in the second key-value cache corresponding to the dynamic attention head is divided into several image blocks; based on the degree of information change of the same image block between adjacent frames, the key-value cache corresponding to the image block that meets the preset conditions is retained to obtain the dynamic head compression cache. The current video segment is generated based on the static header compression cache and the dynamic header compression cache.

2. The method of claim 1, wherein, The division of attention heads in the autoregressive video diffusion model, at least determining static attention heads and dynamic attention heads, includes: Before performing inference using the autoregressive video diffusion model, the attention distribution of each attention head in the autoregressive video diffusion model is obtained; Based on the attention distribution, the corresponding attention heads are divided into static attention heads or dynamic attention heads.

3. The method of claim 2, wherein, The step of obtaining the attention distribution of each attention head in the autoregressive video diffusion model includes: The attention quality of each attention head assigned to different frame regions along the key dimension is statistically analyzed. Calculate the proportion of attention distribution for each attention head in each frame region, and use it as the attention distribution.

4. The method of claim 3, wherein, The frame region includes at least: the current video segment, the transition anchor frame, and the full attention window; The step of classifying the corresponding attention heads into static attention heads or dynamic attention heads based on the attention distribution includes: When the attention head corresponding to the attention distribution ratio indicator focuses more than a preset percentage of attention on the current video segment and the transition anchor frame, the attention head is determined to be the static attention head; When the attention distribution ratio indicates that the attention of the attention head is evenly distributed on the complete attention window, the attention head is determined to be the dynamic attention head.

5. The auto-regressive video diffusion model key-value cache compression method of claim 1, wherein, The step of performing structured pruning on the first key-value cache corresponding to the static attention header to obtain a compressed static header cache includes: In the first key-value cache, retrieve the key-value cache corresponding to the current video segment; The key-value cache corresponding to the sink frame and the key-value cache corresponding to the transition anchor frame in the historical video segment adjacent to the current video segment are determined as the first historical key-value cache. In the key-value caches corresponding to all historical video segments, pruning is performed on the key-value caches other than the first historical key-value cache.

6. The auto-regressive video diffusion model key-value cache compression method of claim 1, wherein, The step of retaining key-value caches corresponding to image blocks that meet preset conditions based on the degree of information change of the same image block between adjacent frames to obtain a dynamic header compression cache includes: Calculate the key state similarity of the same image patch between adjacent frames; The degree of information change is determined based on the key state similarity. When the degree of information change meets a preset condition, the corresponding image block is determined as an image block to be retained, and the key value cache corresponding to the image block to be retained is retained.

7. The method of claim 6, wherein, The calculation of the key state similarity of the same image patch between adjacent frames includes: Obtain the key states of the first layer of the diffusion Transformer in the autoregressive video diffusion model; Based on the key states of the first layer of diffusion Transformer, the cosine similarity of the same image patch between adjacent frames is calculated as the key state similarity.

8. The auto-regressive video diffusion model key-value cache compression method of claim 6, wherein, The step of determining the corresponding image block as an image block to be retained when the degree of information change meets a preset condition includes: The image blocks whose key state similarity is lower than a preset threshold are identified as the image blocks to be retained. Alternatively, the image blocks can be sorted from low to high according to the key state similarity, and a preset number of the image blocks can be selected from low to high to determine the image blocks to be retained.

9. The auto-regressive video diffusion model key-value cache compression method of claim 1, wherein, The step of generating the current video segment based on the static header compression cache and the dynamic header compression cache includes: All the static header compression caches are merged with all the dynamic header compression caches to obtain a compressed key-value cache; When the autoregressive video diffusion model performs inference, it performs attention calculation on the compressed key-value cache using the query vector of the current video segment to generate the current video segment.

10. An autoregressive video diffusion model key-value cache compression system, comprising: include: The partitioning module is used to partition the attention heads in the autoregressive video diffusion model, at least determining the static attention head and the dynamic attention head; The first compression module is used to perform structured pruning on the first key-value cache corresponding to the static attention head to obtain a static head compressed cache; The second compression module is used to divide each frame of the historical video segment in the second key-value cache corresponding to the dynamic attention head into several image blocks, and retain the key-value cache corresponding to the image block that meets the preset conditions according to the degree of information change of the same image block between adjacent frames, so as to obtain the dynamic head compression cache. The generation module is used to generate the current video segment based on the static header compression cache and the dynamic header compression cache.

Images