A video generation method and system based on three-dimensional sparse attention

Abstract

The application discloses a video generation method and system based on three-dimensional sparse attention, and belongs to the field of video generation.The application comprises the following steps: carrying out three-dimensional blocking on input video features according to the time dimension block size and the space dimension block number, and rearranging and indexing each three-dimensional sub-block; adopting a block-level Top-K attention mechanism to select key blocks for each query only; carrying out attention calculation on the query, key and value features of the key blocks, and normalizing and outputting the calculation results; using FlashAttention to perform efficient variable-length attention calculation on the rearranged query, key and value features and restore the original sequence; and generating video output features through residual connection and a feedforward network.The application significantly reduces the long-sequence video attention calculation complexity and the memory occupation, improves the spatiotemporal dependence modeling efficiency, and is suitable for large-scale video generation and understanding tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video generation, in particular to a video generation system based on time-space three-dimensional partitioning and block-level sparse attention, and more particularly to a video generation method and system based on three-dimensional sparse attention. BACKGROUND

[0002] In the field of video understanding and generation, the attention mechanism in the Transformer model is crucial for capturing spatio-temporal dependencies. Among them, the global attention mechanism can most comprehensively model long-range spatio-temporal dependencies by calculating the interaction between all word tokens in the video sequence. However, its inherent defects are also very significant: the computational complexity and memory consumption both grow quadratically with the sequence length, i.e. when processing high-resolution or long-duration videos, this quadratic complexity will cause serious performance and memory bottlenecks, greatly limiting its application in real-world scenarios.

[0003] In order to alleviate the computational pressure of global attention, local attention methods such as sliding window attention have emerged. This method divides the input sequence into fixed-size local windows and only performs self-attention calculations within each window. Although this effectively improves computational efficiency, it comes at the cost of sacrificing the model's receptive field. The attention range is strictly limited within the isolated window, resulting in the model's inability to effectively capture long-range dependencies across window boundaries, and the quality of generated videos in complex dynamic scenarios requiring global information understanding will therefore decrease significantly.

[0004] In addition, some existing sparse attention techniques also have design flaws in practice. These methods usually only apply a pre-set sparse pattern during the model inference stage to speed up computation, but still rely on dense global attention during the training stage. This inconsistency between training and inference stages prevents the model from optimizing the sparse structure end-to-end. As a direct consequence, when switching to sparse mode during inference, the model's performance will suffer a significant quality loss compared to the baseline using global attention.

[0005] In summary, existing technical routes face a dilemma: either bear the high computational cost of global attention, or accept the performance loss of local or sparse attention methods due to limited receptive field and training-inference inconsistency. Therefore, there is an urgent need for a new type of sparse attention mechanism that can effectively model long-range spatio-temporal dependencies while maintaining low computational and memory overhead, and achieving end-to-end training optimization, thereby achieving a balance between performance and efficiency in high-load video processing tasks. SUMMARY

[0006] To address the shortcomings of existing technologies, the present invention aims to provide a video generation method and system based on three-dimensional sparse attention.

[0007] The objective of this invention is achieved through the following technical solution: a video generation method based on three-dimensional sparse attention, comprising the following steps:

[0008] Step 1: Divide the feature tensor of the input video into time dimension according to the preset frame block size, and divide it into three-dimensional sub-blocks according to the row and column grid in the spatial dimension. Perform feature rearrangement on the three-dimensional sub-blocks to generate the rearranged query and key features of each sub-block.

[0009] Step 2: Calculate the attention score between the rearranged query and the key features of each sub-block, and select the sub-block with the highest attention score for each query as the key sub-block;

[0010] Step 3: Based on the query, key, and value features of the key sub-blocks, call the attention interface to perform sparse attention calculation and output the aggregated feature tensor;

[0011] Step 4: Perform maximum value alignment on the aggregated feature tensor and normalize it online to obtain the normalized feature tensor;

[0012] Step 5: Rearrange the normalized feature tensors in reverse to restore the original spatiotemporal order and fuse them into the final video features.

[0013] Furthermore, step one is implemented through the following sub-steps:

[0014] (1.1) Time block: The continuous frame sequence with a total of F frames is divided into (F+bf-1) / / bf time blocks according to the preset frame block size bf. Each time block contains a maximum of bf frames, and the number of frames in the last time block can be less than bf.

[0015] (1.2) Spatial partitioning: For each time block with height and width H and W, it is divided into nh×nw grids according to the preset number of partition blocks. Each grid contains H / / nh rows and W / / nw columns. The number of rows and columns of the last row and the last column block can be different.

[0016] (1.3) Intra-block index calculation: For each position (f, h, w) within a 3D sub-block, calculate its linear offset within the block;

[0017] (1.4) Reordered index generation: Generate a global reordered index for each position by accumulating the prefix sum based on the size of each sub-block;

[0018] (1.5) Feature rearrangement: The original spatiotemporal features are rearranged and stored according to the rearrangement index, so that the features of the same sub-block are continuous in storage.

[0019] Furthermore, step two is implemented through the following sub-steps:

[0020] (2.1) Perform average pooling on the key features belonging to the same sub-block to obtain the block-level representation of each sub-block;

[0021] (2.2) Perform matrix multiplication between the query and the block-level representation of each sub-block to obtain the score of each query for each sub-block, with a size of [num_query, num_heads, num_block];

[0022] (2.3) Perform a topk operation on the score tensor in the sub-block dimension to obtain the k sub-block indexes with the highest scores for each query.

[0023] Furthermore, step three is implemented through the following sub-steps:

[0024] (3.1) Query copying: Copy each element in the flattened query sequence K times to prepare for parallel computation of K key sub-blocks;

[0025] (3.2) Feature arrangement: Based on the selected key sub-block index, the copied query, the corresponding key and value features are arranged to form a format that adapts to the input of the variable length attention interface;

[0026] (3.3) Calling Variable Length Attention: Input the sorted query, key, value and sub-block length information into the variable length attention interface to complete the sparse attention calculation and output the aggregated feature tensor.

[0027] Furthermore, step four is implemented through the following sub-steps:

[0028] (4.1) Maximum value alignment: Subtract the maximum score at each query position from the attention score at that position to prevent exponential overflow;

[0029] (4.2) Exponent calculation: Perform an exponentiation operation on the aligned scores to obtain the non-normalized weights;

[0030] (4.3) Accumulated normalization: sum all the index weights for the same query position as the denominator, and divide each index weight by the denominator to obtain the normalized weight;

[0031] (4.4) Weighted output: Multiply the normalized weights by the corresponding feature values ​​and sum them up to obtain the weighted attention output.

[0032] Furthermore, step five also includes: adding the normalized feature tensor to the original features before rearrangement through residuals, and then processing it through a feedforward neural network to generate the final video features.

[0033] The present invention also provides a video generation system based on three-dimensional sparse attention, comprising:

[0034] The 3D segmentation and feature rearrangement module is used to divide the feature tensor of the input video into time dimension according to a preset frame block size, and divide it into grids in the spatial dimension according to row and column grids to form 3D sub-blocks. The module then performs feature rearrangement on the 3D sub-blocks to generate the rearranged query and key features of each sub-block.

[0035] The attention score calculation module is used to calculate the attention score between the rearranged query and the key features of each sub-block, and select the sub-block with the highest attention score as the key sub-block for each query;

[0036] The sparse attention computation module is used to call the attention interface to perform sparse attention computation based on the query, key, and value features of the key sub-block, and output the aggregated feature tensor.

[0037] The normalization module is used to perform maximum value alignment on the aggregated feature tensor and perform online normalization to obtain the normalized feature tensor.

[0038] The feature inverse rearrangement and fusion module is used to inverse rearrange the normalized feature tensors back to their original spatiotemporal order and fuse them into the final video features.

[0039] Furthermore, the feature inverse rearrangement and fusion module includes a feedforward neural network computation submodule, which adds the normalized feature tensor to the original video features before rearrangement through residuals, and then processes the data through a feedforward neural network to generate the final video features.

[0040] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the video generation method based on three-dimensional sparse attention.

[0041] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video generation method based on three-dimensional sparse attention.

[0042] The beneficial effects of this invention are as follows: By dividing video features into sub-blocks according to temporal and spatial dimensions, performing block-level Top-K filtering, variable-length attention calculation, and online normalization and inverse rearrangement operations, this invention achieves efficient sparse attention calculation and long-sequence video generation. This invention significantly reduces the computational complexity and memory usage of attention calculations for long-sequence videos, while balancing global spatiotemporal dependencies and efficient computation, making it suitable for large-scale, high-resolution video generation and understanding tasks. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A flowchart of a video generation method based on three-dimensional sparse attention provided by the present invention;

[0045] Figure 2 The flowchart of the sparse attention algorithm provided by this invention. Detailed Implementation

[0046] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0047] like Figure 1 As shown, this embodiment of the invention provides a video generation method based on three-dimensional sparse attention, including the following steps:

[0048] Step 1, 3D Blocking and Rearrangement: The feature tensor of the input video is divided into time dimensions according to a preset frame block size, and then divided into grids in the spatial dimension according to row and column grids to form 3D sub-blocks. The features of the 3D sub-blocks are rearranged to generate the rearranged query and key features of each sub-block.

[0049] Step 2, Block-level Top-K Score and Filtering: Calculate the attention score of the rearranged query and the key features of each sub-block, and select the sub-block with the highest attention score for each query as the key sub-block;

[0050] Step 3: Variable-length sparse attention computation: Based on the query, key, and value features of the key sub-block, call the attention interface to perform sparse attention computation and output the aggregated feature tensor;

[0051] Step 4, Online Normalization: Perform maximum value alignment on the aggregated feature tensors and perform online normalization to obtain the normalized feature tensors;

[0052] Step 5, Inverse Rearrangement and Fusion: The normalized feature tensors are inversely rearranged to restore the original spatiotemporal order and fused into the final video features.

[0053] In a preferred embodiment, step one specifically includes the following sub-steps:

[0054] 1.1 First, obtain the feature tensor of the original video, assuming a total number of frames F and a resolution of height H and width W for each frame. In the time dimension, based on the preset frame block size... The video frames are divided into uniform blocks, meaning the entire sequence is divided into several time blocks. The number of time blocks is calculated using the following formula:

[0055] ;

[0056] Each time block contains consecutive... The last time block of the frame may be insufficient. Frames, but ensure that all frames are covered.

[0057] Specifically, the index range of the frames contained in the i-th time block is:

[0058] ;

[0059] Through the above calculations, regardless of the change in video length, all frames can be completely segmented without any omissions.

[0060] 1.2. Regarding the spatial dimension, each frame of the image is divided according to preset grid parameters. lines and The column is further subdivided into multiple sub-blocks. The height and width of each grid block are spatially divided equally using the following formulas:

[0061] ;

[0062] The specific pixel range covered by the grid in the j-th row and k-th column is as follows:

[0063] ;

[0064] The above calculations not only ensure the continuity of the height and width divisions, but also take into account the automatic adaptation of boundary blocks caused by divisibility relationships. In this case, the actual size of the last row or the last column may be smaller than the size of the uniform block.

[0065] 1.3 After completing the time and space partitioning, generate three-dimensional sub-blocks. Where i, j, and k represent their block indices on F, H, and W, respectively, and the number of elements contained in each is calculated using the following formula:

[0066] ;

[0067] in, This indicates the size of the sub-block in dimension F. This indicates the size of the sub-block in the H dimension. This indicates that the sub-block is in W.

[0068] Furthermore, to facilitate subsequent tensor operations, the set of position index prefixes of the first element in the global tensor is calculated sequentially for all sub-blocks. .

[0069] For specific locations within a sub-block, the timing offset, row offset, and column offset are defined as follows:

[0070] .

[0071] The linear index of this position in the global flat storage structure is:

[0072] .

[0073] The final global index of each feature point is:

[0074] .

[0075] The rearrangement operation, following the indexing rules described above, re-stores all feature points of the video, ensuring that all features within each block are tightly and continuously connected in the storage structure.

[0076] In a preferred embodiment, after completing the three-dimensional segmentation and rearrangement in step one, the system will perform a three-dimensional sub-block... Key features after top rearrangement and query features (in Indicates the attention head index. (Indicating a query location index) to perform block-level filtering, step two specifically includes the following sub-steps:

[0077] 2.1 Calculation of Block Mean Key

[0078] For the All positions within the sub-block key vector First, calculate the bond mean of the sub-block. The formula is:

[0079] ;

[0080] in, Indicates a sub-block The set of all position indices within. For sub-blocks The total number of elements.

[0081] 2.2 Score Matrix Calculation

[0082] Based on query features Average of each sub-block bond Perform a dot product operation to obtain the score tensor. This simplifies to mapping sub-block indices to a single dimension. :

[0083] .

[0084] 2.3 Top-K Screening

[0085] For each head h and query position i, a score vector Perform a descending sort on the block dimension (i.e., the m-dimensional index) to obtain the sorted index sequence. ,from Extract the first K indices to form a key sub-block index set:

[0086] ;

[0087] in, This refers to the K most important 3D sub-block numbers under the h-th head of the query. Subsequent sparse attention calculations are only performed on these selected sub-blocks, thereby significantly reducing the amount of computation.

[0088] Through the above three sub-steps, block-level score calculation and Top-K filtering are completed for all query positions and attention heads, providing a concise and high-quality sub-block index input for the variable-length sparse attention calculation in subsequent steps.

[0089] As a preferred embodiment, after completing the Top-K sub-block selection, step three involves rearranging the input format to a form acceptable to FlashAttention and then feeding it into FlashAttention to complete the attention operation in one go. Figure 2 As shown, specifically:

[0090] First, the key-value pairs of all spatiotemporal locations within sub-block m under the h-th attention head are considered as the KV sequence of that sub-block. The length of this sequence is equal to the number of elements in the sub-block. Second, all query vectors Q[h, i] selected in step two are collected to form the Q sequence of that sub-block. Since the number of times the same sub-block is selected may vary at different query positions, the length of the Q sequence is equal to the total number of queries that are not selected for that sub-block during the filtering process. In this way, the attention calculations for the same or different sub-blocks by different queries can be completed in parallel in a single round of calls, achieving sparsity in computation and storage.

[0091] As a preferred embodiment, step four specifically includes: This step aims to normalize the unnormalized scores corresponding to each query position in the sparse attention calculation, so as to ensure the numerical stability of the calculation and effectively merge the partial sums generated by local calculations, and finally obtain the correct weighted feature output of each query head.

[0092] After the previous FlashAttention calculation, we obtain multiple partial outputs for each query position and attention head distribution across several selected sub-blocks, along with the logarithmic-exponential sum of the corresponding partial outputs. Since the same query position typically focuses on multiple sub-blocks simultaneously, this results in multiple partial outputs and corresponding partial LSEs. These must be precisely merged and normalized to restore the globally correct attention weight distribution and weighted features. This merging process is the core task of this step.

[0093] First, for the input attention score vector The function calculates the maximum score for all elements in the same query header group. :

[0094] .

[0095] By subtracting the maximum value from each element, the scores are aligned uniformly, thus avoiding exponential overflow:

[0096] .

[0097] Next, the exponent is calculated element-wise for all aligned scores to obtain the non-normalized attention weights:

[0098] .

[0099] Then, the exponential weights belonging to the same query header group are summed, and the normalized denominator for that group is calculated:

[0100] .

[0101] The normalized weights are:

[0102] .

[0103] Next, each weight will be associated with its corresponding value feature. Multiply each element and sum them to generate the weighted output of the query header group:

[0104] .

[0105] It is worth noting that the partial sums generated in the first few steps of sparse attention computation are obtained by weighting different sub-blocks of the same query header separately. The online normalization step achieves effective integration of all partial outputs by accumulating the weighted results in groups, ensuring that the correct global attention weighted output is obtained.

[0106] In a preferred embodiment, step five specifically includes the following sub-steps:

[0107] After completing the aforementioned online normalization step, the normalized sub-block output features are obtained. To restore the original spatiotemporal structure of the video, an inverse rearrangement operation must be performed, mapping the features in the sub-block back to their corresponding spatiotemporal coordinates according to the initial rearrangement rules. Let the normalized sub-block output be... It contains Elements, inverse rearrangement via a mapping function Accurately map each element back to its spatiotemporal index in the original video tensor. ,Right now:

[0108] ;

[0109] Wherein, mapping function This refers to the three-dimensional sub-block numbering rules in step one, and the index. This represents the linear position within the sub-block.

[0110] Subsequently, to enhance the stability and transitivity of feature representation, the output feature O after inverse rearrangement is residually connected to the original input feature X, and the following is calculated:

[0111] ;

[0112] This residual connection effectively alleviates the gradient vanishing problem in deep networks and promotes the effective transmission of feature information.

[0113] Finally, the fused feature F is input into the feedforward neural network module to complete the nonlinear transformation and extraction of complex features:

[0114] ;

[0115] in, and These are the learnable parameters of FFN. It is a non-linear activation function (such as ReLU). This layer helps to improve the expressiveness of spatiotemporal features and the generalization performance of the model.

[0116] In summary, the inverse rearrangement and fusion steps take into account both the restoration of spatial temporal structure and the enhancement of deep features, ensuring that the final video features can faithfully reflect the complex spatiotemporal information of the original video, and providing high-quality representations for downstream video understanding and generation tasks.

[0117] The present invention also provides a video generation system based on three-dimensional sparse attention, comprising:

[0118] The 3D segmentation and feature rearrangement module is used to divide the feature tensor of the input video into time dimension according to a preset frame block size, and divide it into grids in the spatial dimension according to row and column grids to form 3D sub-blocks. The module then performs feature rearrangement on the 3D sub-blocks to generate the rearranged query and key features of each sub-block.

[0119] The attention score calculation module is used to calculate the attention score between the rearranged query and the key features of each sub-block, and select the sub-block with the highest attention score as the key sub-block for each query;

[0120] The sparse attention computation module is used to call the attention interface to perform sparse attention computation based on the query, key, and value features of the key sub-block, and output the aggregated feature tensor.

[0121] The normalization module is used to perform maximum value alignment on the aggregated feature tensor and perform online normalization to obtain the normalized feature tensor.

[0122] The feature inverse rearrangement and fusion module is used to inverse rearrange the normalized feature tensors back to their original spatiotemporal order and fuse them into the final video features.

[0123] The implementation process of the functions and roles of each module in the above system is detailed in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0124] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the video generation method based on three-dimensional sparse attention.

[0125] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video generation method based on three-dimensional sparse attention.

[0126] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0127] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.

[0128] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A video generation method based on three-dimensional sparse attention, characterized in that, Includes the following steps: Step 1: Divide the feature tensor of the input video into time dimension according to the preset frame block size, and divide it into three-dimensional sub-blocks according to the row and column grid in the spatial dimension. Perform feature rearrangement on the three-dimensional sub-blocks to generate the rearranged query and key features of each sub-block. Step 2: Calculate the attention score between the rearranged query and the key features of each sub-block, and select the sub-block with the highest attention score for each query as the key sub-block; Step 3: Based on the query, key, and value features of the key sub-blocks, call the attention interface to perform sparse attention computation and output the aggregated feature tensor; including: (3.1) Copy each element in the flattened query sequence K times to prepare for parallel computation of K key sub-blocks; (3.2) Based on the selected key sub-block index, arrange the copied query, the corresponding key and value features to form a format that adapts to the input of the variable-length attention interface; (3.3) Input the sorted query, key, value and sub-block length information into the variable length attention interface to complete the sparse attention calculation and output the aggregated feature tensor; Step 4: Perform maximum alignment and online normalization on the aggregated feature tensors to obtain normalized feature tensors; including: (4.1) Subtract the maximum score for each query position from the attention score for that position to prevent exponential overflow; (4.2) Perform an exponential operation on the aligned scores to obtain the non-normalized weights; (4.3) Sum all the index weights for the same query position as the denominator, and divide each index weight by the denominator to obtain the normalized weight; (4.4) Multiply the normalized weights by the corresponding feature values ​​and sum them up to obtain the weighted attention output; Step 5: Rearrange the normalized feature tensors back to their original spatiotemporal order and fuse them into the final video features.

2. The video generation method based on three-dimensional sparse attention according to claim 1, characterized in that, Step one is achieved through the following sub-steps: (1.1) Divide the continuous frame sequence with a total of F frames into (F+bf-1) / / bf time blocks according to the preset frame block size bf. Each time block contains a maximum of bf frames, and the number of frames in the last time block can be less than bf. (1.2) For each time block with height and width H and W, it is divided into nh×nw grids according to the preset number of blocks with height and width. Each grid contains H / / nh rows and W / / nw columns. The number of rows and columns of the last row and the last column block may not be equal. (1.3) Calculate the linear offset of each position (f, h, w) within the three-dimensional sub-block, where f, h, w represent the sub-block index in the three dimensions of frame number, height, and width, respectively; (1.4) Generate a global rearranged index for each position by accumulating the prefix sum based on the size of each sub-block; (1.5) Rearrange the original spatiotemporal features according to the rearrange index so that the features of the same sub-block are continuous in storage.

3. The video generation method based on three-dimensional sparse attention according to claim 1, characterized in that, Step two is achieved through the following sub-steps: (2.1) Perform average pooling on the key features belonging to the same sub-block to obtain the block-level representation of each sub-block; (2.2) Perform matrix multiplication between the query and the block-level representation of each sub-block to obtain the score S of each query for each sub-block; (2.3) Perform a topk operation on the score tensor in the sub-block dimension to obtain the index of the k sub-blocks with the highest scores for each query.

4. The video generation method based on three-dimensional sparse attention according to claim 1, characterized in that, Step five further includes: adding the normalized feature tensor to the original video features before rearrangement through residuals, and then processing it through a feedforward neural network to generate the final video features.

5. A video generation system based on three-dimensional sparse attention, characterized in that, include: The 3D segmentation and feature rearrangement module is used to divide the feature tensor of the input video into time dimension according to a preset frame block size, and divide it into grids in the spatial dimension according to row and column grids to form 3D sub-blocks. The module then performs feature rearrangement on the 3D sub-blocks to generate the rearranged query and key features of each sub-block. The attention score calculation module is used to calculate the attention score between the rearranged query and the key features of each sub-block, and select the sub-block with the highest attention score as the key sub-block for each query; The sparse attention computation module is used to call the attention interface to perform sparse attention computation based on the query, key, and value features of the key sub-blocks, and output the aggregated feature tensor; including: Each element in the flattened query sequence is copied K times to prepare for parallel computation of the K key sub-blocks; Based on the selected key sub-block index, the copied query, corresponding key and value features are arranged to form a format that adapts to the input of the variable-length attention interface; Input the sorted query, key, value and sub-block length information into the variable-length attention interface to complete the sparse attention calculation and output the aggregated feature tensor; The normalization module is used to perform maximum value alignment and online normalization on the aggregated feature tensors to obtain normalized feature tensors; it includes: Subtract the maximum score for each query location from the attention score for that location to prevent exponential overflow; Perform an exponential operation on the aligned scores to obtain the non-normalized weights; The normalized weights are obtained by summing all the exponential weights for the same query position and using the sum as the denominator, and then dividing each exponential weight by the denominator. The normalized weights are multiplied by their corresponding feature values ​​and summed to obtain the weighted attention output. The feature inverse rearrangement and fusion module is used to inverse rearrange the normalized feature tensors back to their original spatiotemporal order and fuse them into the final video features.

6. The video generation system based on three-dimensional sparse attention according to claim 5, characterized in that, The feature inverse rearrangement and fusion module includes a feedforward neural network computation submodule, which adds the normalized feature tensor to the original video features before rearrangement through residuals, and then processes them through a feedforward neural network to generate the final video features.

7. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the video generation method based on three-dimensional sparse attention as described in any one of claims 1-4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the video generation method based on three-dimensional sparse attention as described in any one of claims 1-4.

Images