Video generation method and apparatus

Abstract

Embodiments of the present application provide a video generation method and device, computer equipment, computer readable storage medium and computer program product, belonging to the field of video processing, comprising: dividing a video token sequence into multiple sequence fragments; distributing the multiple sequence fragments to multiple GPUs for processing; respectively aggregating local query matrices, local key matrices and local value matrices in multiple groups of local matrices; performing space-time position coding on a global query matrix and a global key matrix; dividing the global value matrix and the coded global query matrix and the coded global key matrix according to the attention head dimension, and distributing the divided matrices to the corresponding GPUs; aggregating the attention calculation results of each GPU; and generating multiple target video frames corresponding to multiple initial video frames according to the aggregated attention calculation results. The technical solution of the embodiments of the present application can improve the upper limit of the inference length and the model scalability of video generation.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of artificial intelligence technology, video generation models have shown great potential in the field of content creation. These video generation models can use self-attention mechanisms to handle the spatiotemporal dependencies in token sequences to generate highly consistent video content. However, as video length and resolution increase, the length of the video token sequence also grows rapidly, exceeding the limit of the video memory resources of a single GPU, making it impossible to complete the video generation inference.

[0003] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention

[0004] This application provides a video generation method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems mentioned above.

[0005] One aspect of this application provides a video generation method, the method comprising:

[0006] The video token sequence is divided into multiple sequence segments, and the video token sequence is formed based on multiple initial video frames to be processed. The multiple sequence slices are distributed to multiple GPUs for processing to obtain multiple sets of local matrices. Each GPU corresponds to one sequence slice and one set of local matrices. A set of local matrices includes a local query matrix, a local key matrix, and a local value matrix. The local query matrix, local key matrix, and local value matrix in the multiple sets of local matrices are aggregated to obtain the global query matrix, global key matrix, and global value matrix. Spatiotemporal location encoding is performed on the global query matrix and the global key matrix; The global value matrix, the encoded global query matrix, and the encoded global key matrix are divided according to the attention head dimension, and the divided matrices are distributed to the corresponding GPUs so that attention calculations can be performed by the corresponding GPUs to obtain the attention calculation results. Aggregate the attention calculation results from each GPU; and Multiple target video frames are generated based on the aggregated attention calculation results, corresponding to the multiple initial video frames.

[0007] Optionally, the video token sequence includes multiple tokens; The video token sequence is divided into multiple sequence fragments, including: The number of tokens in each sequence slice is determined based on the number of multiple sequence slices and the number of multiple tokens; The video token sequence is divided into multiple sequence segments based on the number of tokens in each sequence segment.

[0008] Optionally, the plurality of initial video frames are multiple adjacent video frames in an initial video frame sequence; Spatiotemporal location encoding of the global query matrix and global key matrix includes: Based on the global time index, height information, and width information of each token, retrieve the trigonometric function values ​​corresponding to each token from the local cache of each GPU; and Based on the trigonometric function values ​​corresponding to each token, the global query matrix and global key matrix are spatiotemporally encoded. The global time index represents the position of the initial video frame corresponding to the token in the initial video frame sequence.

[0009] Optionally, each GPU corresponds to at least one attention head; The global value matrix, encoded global query matrix, and encoded global key matrix are divided according to the attention head dimension, and the divided matrices are distributed to the corresponding GPUs, including: Based on at least one attention head corresponding to each GPU, the global value matrix, the encoded global query matrix, and the encoded global key matrix are each divided into multiple sub-matrices; each sub-matrix corresponds to at least one attention head of the same GPU; and Distribute each of the multiple sets of sub-matrices to the corresponding GPU.

[0010] Optionally, each set of submatrices includes a subquery matrix, a subkey matrix, and a subvalue matrix, and each GPU is used to update the key cache and value cache based on the obtained subkey matrix and subvalue matrix.

[0011] Optionally, attention calculations are performed using the corresponding GPU to obtain the attention calculation results, including: The attention calculation results of the corresponding attention head are obtained by performing attention calculations on the corresponding GPU based on its updated key cache and value cache, as well as the corresponding subquery matrix.

[0012] Optionally, multiple target video frames corresponding to the multiple initial video frames are generated based on the aggregated attention calculation results, including: The aggregated attention calculation results are segmented by each GPU to obtain multiple segmented attention calculation results; each segmented attention calculation result corresponds to a sequence slice processed by one GPU. Each GPU performs cross-attention calculations based on a pre-set text description and the attention calculation results of the segmented data corresponding to that GPU, to generate the video generation result of the sequence segments corresponding to that GPU; and The video generation results of the sequence segments corresponding to each GPU are aggregated to obtain the multiple target video frames.

[0013] Another aspect of this application provides a video generation method, including: The video token sequence is divided into multiple sequence segments. The video token sequence is formed based on multiple initial video frames to be processed. The multiple initial video frames are multiple adjacent video frames in the initial video frame sequence. The multiple sequence slices are distributed to multiple GPUs for processing to obtain multiple sets of local matrices. Each GPU corresponds to one sequence slice and one set of local matrices. A set of local matrices includes a local query matrix, a local key matrix, and a local value matrix. In each GPU, spatiotemporal position encoding is performed on the local query matrix and local key matrix based on the position of the first frame in the initial video frame sequence. In multiple GPUs, cross-GPU data redistribution is performed on the local value matrix, the encoded local query matrix, and the encoded local key matrix, so that each GPU obtains the matrix to be computed for attention computation. The attention calculation result is obtained by performing attention calculation on each GPU based on the obtained matrix to be calculated. Aggregate the attention calculation results from each GPU; and Multiple target video frames are generated based on the aggregated attention calculation results, corresponding to the multiple initial video frames.

[0014] Optionally, in multiple GPUs, cross-GPU data redistribution is performed on the respective local value matrices, encoded local query matrices, and encoded local key matrices, including: In each GPU, the local value matrix, the encoded local query matrix, and the encoded local key matrix are divided into multiple data slices according to the attention head dimension and distributed to the corresponding other GPUs. The data slices received from other GPUs are then concatenated according to the sequence dimension so that each GPU obtains a matrix to be computed for attention calculation.

[0015] Optionally, each sequence shard includes multiple tokens; Based on the position of the first frame in the initial video frame sequence, spatiotemporal position encoding is performed on their respective local query matrices and local key matrices, including: Based on the position of the first frame in the initial video frame sequence, the global time index of each token is determined. Based on the global time index, height information, and width information of each token, retrieve the trigonometric function values ​​corresponding to each token from the local cache of each GPU; Based on the trigonometric function values ​​corresponding to each token, the local query matrix and local key matrix are spatiotemporally encoded. The global time index represents the position of the initial video frame corresponding to the token in the initial video frame sequence.

[0016] Another aspect of this application provides a video generation apparatus, the apparatus comprising: The segmentation module is used to divide the video token sequence into multiple sequence segments. The video token sequence is formed based on multiple initial video frames to be processed. The processing module is used to distribute the multiple sequence slices to multiple GPUs for processing, so as to obtain multiple sets of local matrices. One GPU corresponds to one sequence slice and one set of local matrices. One set of local matrices includes a local query matrix, a local key matrix and a local value matrix. The first aggregation module is used to aggregate the local query matrix, local key matrix and local value matrix in the multiple sets of local matrices respectively to obtain the global query matrix, global key matrix and global value matrix; The encoding module is used to perform spatiotemporal position encoding on the global query matrix and the global key matrix; The distribution module is used to divide the global value matrix, the encoded global query matrix, and the encoded global key matrix according to the attention head dimension, and distribute the divided matrices to the corresponding GPUs so that attention calculations can be performed by the corresponding GPUs to obtain the attention calculation results. The second aggregation module is used to aggregate the attention calculation results of each GPU; and The generation module is used to generate multiple target video frames corresponding to the multiple initial video frames based on the aggregated attention calculation results.

[0017] Another aspect of this application provides a video generation apparatus, the apparatus comprising: The segmentation module is used to divide the video token sequence into multiple sequence segments. The video token sequence is formed based on multiple initial video frames to be processed. The multiple initial video frames are multiple adjacent video frames in the initial video frame sequence. The processing module is used to distribute the multiple sequence slices to multiple GPUs for processing, so as to obtain multiple sets of local matrices. One GPU corresponds to one sequence slice and one set of local matrices. One set of local matrices includes a local query matrix, a local key matrix and a local value matrix. The encoding module is used in each GPU to perform spatiotemporal position encoding on the local query matrix and local key matrix based on the position of the first frame in the initial video frame sequence among the plurality of initial video frames. The redistribution module is used to perform cross-GPU data redistribution on the local value matrix, the encoded local query matrix, and the encoded local key matrix of each GPU, so that each GPU can obtain the matrix to be computed for attention calculation. The computation module is used to perform attention calculations on each GPU based on the obtained matrix to be calculated in order to obtain the attention calculation results; The aggregation module is used to aggregate the attention calculation results from each GPU; and The generation module is used to generate multiple target video frames corresponding to the multiple initial video frames based on the aggregated attention calculation results.

[0018] Another aspect of this application provides a computer device, including: At least one processor; and A memory that is communicatively connected to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0019] Another aspect of this application provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method described above.

[0020] Another aspect of this application provides a computer program product including a computer program that, when executed by a processor, implements the method described above.

[0021] The technical solution adopted in this application embodiment may have the following advantages: by dividing the large-scale video token sequence into segments along the sequence dimension and distributing them to multiple GPUs for parallel processing, and performing distributed computing according to the attention head dimension after spatiotemporal position encoding of the global matrix, the memory and computing load of long sequence video generation are distributed to multiple GPUs, breaking through the physical limitation of single GPU computing resources on video length, realizing efficient inference and generation of ultra-long videos, and improving the upper limit of inference length and model scalability of video generation. Attached Figure Description

[0022] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0023] Figure 1 The diagram illustrates the operating environment of the video generation method according to Embodiment 1 of this application. Figure 2 A flowchart illustrating a video generation method according to Embodiment 1 of this application is shown schematically; Figure 3 Schematic illustration Figure 2 Flowchart of the sub-steps in step S200; Figure 4 Schematic illustration Figure 2 Flowchart of the sub-steps in step S206; Figure 5 Schematic illustration Figure 2 Flowchart of the sub-steps in step S208; Figure 6 Schematic illustration Figure 2 Flowchart of the sub-steps in step S212; Figure 7 The diagram schematically illustrates an exemplary application algorithm diagram of the video generation method according to Embodiment 1 of this application; Figure 8 A flowchart illustrating a video generation method according to Embodiment 2 of this application is shown schematically; Figure 9 Schematic illustration Figure 8 Flowchart of the sub-steps in step S804; Figure 10 The diagram schematically illustrates an exemplary application algorithm diagram of the video generation method according to Embodiment 2 of this application; Figure 11 A block diagram of a video generation apparatus according to Embodiment 3 of this application is shown schematically; Figure 12 A block diagram of a video generation apparatus according to Embodiment 4 of this application is schematically shown; and Figure 13 A schematic diagram of the hardware architecture of a computer device according to Embodiment 3 of this application is shown. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0025] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0026] It should be noted that, in any stage of this application involving the collection, storage, use, transmission, and processing of data, each stage strictly adheres to the laws, regulations, industry standards, and regulatory requirements of the data source, usage location, and relevant countries and regions to ensure the legality and compliance of data activities. In the collection stage, the purpose, method, and scope of collection are clearly communicated to the data subject in a prominent manner. Collection is conducted only after obtaining the data subject's legal authorization, ensuring that the collection process follows the "minimum necessary" principle and does not exceed the scope of data collection. In the storage stage, storage periods are limited, and data is promptly deleted or anonymized / encrypted after the storage purpose is achieved. In the usage stage, a strict data security protection mechanism is implemented, using field-level desensitization technology and processing the original data according to preset desensitization rules. For different types of data, multiple desensitization strategies, such as data generalization, data anonymization, and data encryption, are employed to effectively mitigate the risk of sensitive information leakage and ensure that all data used is securely processed and desensitized, comprehensively protecting the rights and interests of data subjects and data security. In the transmission and processing stages, the confidentiality and security of data are ensured during transmission and processing.

[0027] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.

[0028] First, a definition of the terminology used in this application is provided: GPU (Graphics Processing Unit) is used for image rendering, AI / deep learning, video encoding and decoding, and high-performance computing.

[0029] Token: The smallest unit of text or video data processed by the model.

[0030] Video reasoning refers to the process of using a trained video generation model to gradually generate a sequence of video frames based on input text descriptions, images, or other conditions through iterative denoising calculations.

[0031] Diffusion model: A generative AI model that first adds noise to the original data (such as images and videos) during training until the data becomes pure noise, and then trains a neural network to learn the reverse denoising process to gradually recover clear data from the noise.

[0032] 3D Causal Variational Autoencoder: A video compression model that uses a causal convolutional structure to ensure that the current frame can only rely on information from historical frames during encoding. At the same time, it uses 3D convolutional kernels to jointly model spatiotemporal features, compressing the input video into a compact latent representation.

[0033] Spatiotemporal compression: refers to the technique of downsampling and compressing video data in both time and space dimensions.

[0034] Query matrix: Represents the information requests that need to be followed. Each token matches the keys of other tokens through a query vector to determine which information should be followed.

[0035] Key matrix: Used for retrieval and matching of the query matrix. The similarity between the key vector of each token and the query vector is calculated. The higher the similarity, the more important the token is to the current query.

[0036] Value matrix: Stores the actual information to be conveyed. The attention mechanism performs weighted aggregation on the value matrix based on the degree of matching between the query and the key, and finally outputs the fused feature representation.

[0037] QKV projection: The process of mapping input features into query vector, key vector and value vector through three independent linear transformation matrices.

[0038] 3DRoPE: A three-dimensional rotation position encoding method that decomposes the rotation frequency into three parts in the feature dimension: time, height, and width. These correspond to the position information of the video token in the three dimensions of temporal frame, vertical space, and horizontal space, respectively. The three-dimensional spatial position of the token is jointly encoded by complex multiplication, enabling the model to perceive the spatiotemporal structure of the video data.

[0039] RoPE fundamental frequency: refers to the fundamental frequency parameter used to generate the rotation angle sequence in rotational position coding.

[0040] Causal masking: a mechanism that ensures that sequence generation models can only "see" past and present information and cannot access future information.

[0041] Causal attention computation: an attention mechanism that restricts information to flow only from front to back. When calculating the current position, the model can only access the current and previous position information and cannot see the future content.

[0042] Cross-attention computation: an attention mechanism in which the query vector comes from one modality (such as text), while the key and value vectors come from another modality (such as an image or video). By calculating the similarity between the text query and the visual key, textual semantic information is selectively injected into the visual generation process.

[0043] Host side: refers to the computing side where the CPU and its memory are located. In contrast, the GPU, as the Device side, focuses on large-scale parallel computing. The Host side communicates with the GPU through the PCIe bus, issuing computing instructions and transmitting data.

[0044] Computational flow: A queue of commands executed on the GPU, used to manage the submission and execution order of operations such as kernel computation and memory copying.

[0045] AllToAll: A collection communication operation that divides the data of each participating process (or GPU) into multiple parts and distributes them to all other processes, while receiving the corresponding data parts from other processes.

[0046] Latent: In video generation, it refers to the low-dimensional representation after compression and encoding.

[0047] Secondly, to facilitate understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the relevant technologies are described below: With the development of deep learning technology, video generation techniques based on diffusion models have made significant progress. These video generation models can utilize self-attention mechanisms to handle spatiotemporal dependencies in token sequences, generating highly consistent video content. However, as video duration and resolution increase, the length of the video token sequence also grows rapidly, exceeding the limits of a single GPU's memory resources, making it impossible to complete video generation inference. Furthermore, since rotational position encoding requires complete sequence context information, the complete matrix corresponding to the global token sequence must be synchronized through three cross-GPU communications before position encoding calculations can be performed. This results in strong coupling between computational and communication logic, and the dense communication overhead limits the speed of video inference generation.

[0048] Therefore, this application provides a video generation technology solution. In this technology solution, (1) by dividing a large-scale video token sequence in the sequence dimension and distributing it to multiple GPUs for parallel processing, and after spatiotemporal location encoding of the global matrix, performing distributed computing according to the attention head dimension, the memory and computing load of long sequence video generation are distributed to multiple GPUs, breaking through the physical limitation of single GPU computing resources on video length, realizing efficient inference and generation of ultra-long videos, and improving the upper limit of inference length and model scalability of video generation; (2) by dividing the global value matrix, the encoded global query matrix, and the encoded global key matrix according to the part of attention head that each GPU is responsible for, so that each GPU is responsible for the computing task of a part of the attention head. Thus, after each GPU completes query projection and aggregation of the segmented sequence, it only needs to bear part of the computational load of the attention head, effectively dispersing the computational pressure of the self-attention layer in long video generation, and improving the computational throughput and video generation efficiency; (3) By maintaining key and value caches on each GPU and performing incremental updates based on subkey and subvalue matrices, the attention calculation results of historical frames can be reused in the video block autoregressive generation process, avoiding the repeated encoding of historical global information for each newly generated block, reducing computational redundancy and peak memory usage in long video generation scenarios, while maintaining the consistency of causal attention in the video generation process, and improving the computational efficiency and video quality of video generation; (4) By simultaneously completing the local query, key, and value matrix conversion and reconstruction in the sequence dimension and attention head dimension in a single communication step, the replacement of multiple aggregation communication and segmentation operations is realized, reducing the performance loss caused by cross-GPU communication, and improving the inference speed and efficiency of video generation. See the following text for details.

[0049] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0050] like Figure 1 As shown in the diagram, the operating environment includes: Server 2, Network 6, and Terminal 4, where: Server 2 can consist of one or more computing devices. These computing devices may include virtualized computing instances. Virtualized computing instances may include virtual machines, such as emulations of computer systems, operating systems, servers, etc. The computing devices may load virtual machines based on virtual images and / or other data that define specific software used for emulation (e.g., operating systems, dedicated applications, servers). As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computing devices. A hypervisor can be implemented to manage the use of different virtual machines on the same computing device.

[0051] Server 2 can be configured to communicate with terminal 4, etc., via network 6. Network 6 includes various network devices such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or similar devices. Network 6 may include physical links, such as coaxial cable links, twisted pair cable links, fiber optic links, or combinations thereof, or wireless links, such as cellular links, satellite links, Wi-Fi links, etc.

[0052] Server 2 can provide services such as storage, reading, writing, querying, and deleting, and can also provide video generation services for terminals.

[0053] Terminal 4 can be an electronic device running operating systems such as Windows, Android™, or iOS, such as a workstation, industrial terminal, or PC. Based on these operating systems, various applications can run, such as inputting text descriptions for videos.

[0054] Terminal 4 can provide / configure a user access page, which can be used to control server 2 or upload objects, etc.

[0055] It should be noted that the above-mentioned equipment is exemplary, and the number and type of equipment can be adjusted in different scenarios or according to different needs.

[0056] The technical solutions of this application are described below using server 2 as an exemplary execution entity through multiple embodiments. It should be understood that these embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments described herein.

[0057] Example 1 Figure 2 A flowchart illustrating a video generation method according to Embodiment 1 of this application is shown schematically.

[0058] like Figure 2 As shown, the video generation method may include steps S200~S212, wherein: Step S200: Divide the video token sequence into multiple sequence segments. The video token sequence is formed based on multiple initial video frames to be processed.

[0059] Step S202: Distribute the multiple sequence slices to multiple GPUs for processing to obtain multiple sets of local matrices. Each GPU corresponds to one sequence slice and one set of local matrices. One set of local matrices includes a local query matrix, a local key matrix, and a local value matrix.

[0060] Step S204: Aggregate the local query matrix, local key matrix, and local value matrix in the multiple sets of local matrices to obtain the global query matrix, global key matrix, and global value matrix.

[0061] Step S206: Spatiotemporal position encoding is performed on the global query matrix and the global key matrix.

[0062] Step S208: Divide the global value matrix, the encoded global query matrix, and the encoded global key matrix according to the attention head dimension, and distribute the divided matrices to the corresponding GPUs so that attention calculation can be performed by the corresponding GPUs to obtain the attention calculation results.

[0063] Step S210: Aggregate the attention calculation results of each GPU.

[0064] Step S212: Generate multiple target video frames corresponding to the multiple initial video frames based on the aggregated attention calculation results.

[0065] The video generation method provided in this embodiment divides the video token sequence across multiple GPUs for parallel computation of local queries, key-value matrices, and then uniformly performs spatiotemporal position encoding after aggregating to generate a global matrix. Next, the matrix is ​​re-divided and distributed according to the attention head dimension to perform distributed attention computation, thereby generating the target video frame. By dividing large-scale video token sequences along the sequence dimension and distributing them to multiple GPUs for parallel processing, and performing distributed computation according to the attention head dimension after spatiotemporal position encoding of the global matrix, the memory and computational load for generating long video sequences is distributed across multiple GPUs. This overcomes the physical limitations of single-GPU computing resources on video length, achieving efficient inference and generation of ultra-long videos, and improving the upper limit of inference length and model scalability for video generation.

[0066] The following combination Figure 2 The steps in steps S200 to S212 and other optional steps are described in detail.

[0067] Step S200 The video token sequence is divided into multiple sequence segments, and the video token sequence is formed based on multiple initial video frames to be processed.

[0068] In practical processing, multiple initial video frames can be selected from a sequence of adjacent video frames. Each video frame in the initial video frame sequence can be a noisy video frame randomly generated by a video generation model. After obtaining multiple initial video frames, spatiotemporal compression can be performed on them using a 3D causal variational autoencoder or similar method to form a video token sequence. During compression, the tokens in the video token sequence can be arranged according to frame priority. After obtaining the video token sequence, it can be divided into multiple sequence segments based on the number of tokens and sequence segments.

[0069] In practice, video token sequences can be divided in various ways. The following provides an exemplary division method.

[0070] In an optional embodiment, the video token sequence includes multiple tokens. For example... Figure 3 As shown, step S200 includes: S300, determine the number of tokens in each sequence segment based on the number of multiple sequence segments and the number of multiple tokens.

[0071] S302, the video token sequence is segmented according to the number of tokens in each sequence segment to obtain the plurality of sequence segments.

[0072] For example, if video token sequence A contains 4096 tokens, and this video generation used 4 GPUs, then video token sequence A needs to be divided into 4 sequence fragments. Each GPU's sequence fragment should contain 1024 tokens.

[0073] It should be noted that the above example describes a method of evenly distributing video tokens based on the number of GPUs. In other embodiments, the number of tokens allocated to each GPU in a sequence slice can be dynamically determined by considering the real-time resource status of each GPU (such as remaining video memory capacity, current batch size, etc.). For example, more tokens can be allocated to GPUs with more remaining video memory, and fewer tokens can be allocated to GPUs with less remaining video memory.

[0074] In this embodiment, by dynamically determining the number of tokens in each segment based on the number of sequence segments and the total number of tokens, and then dividing the sequence segments accordingly, the load balancing of the video token sequence among multiple GPUs is achieved. This ensures that the amount of data processed by each GPU is balanced, thereby reducing load skew and resource limitations in sequence parallel computing. This provides a foundation for distributed parallel processing of long video sequences and improves the efficiency and stability of video generation.

[0075] Step S202 The multiple sequence slices are distributed to multiple GPUs for processing to obtain multiple sets of local matrices. Each GPU corresponds to one sequence slice and one set of local matrices. A set of local matrices includes a local query matrix, a local key matrix, and a local value matrix.

[0076] The local query matrix represents what feature information each token in the sequence slice currently being processed by the GPU wants to obtain from other tokens. The local key matrix, as the feature index of the video token, is used to respond to retrieval requests from other tokens, determining whether the semantic features of the current token (such as color, texture, motion pattern) match the query requirements of other tokens. The local value matrix contains the actual latent features of the video token after projection, such as visual content (object shape, color distribution, motion vectors) and semantic information.

[0077] In some embodiments, each GPU can use the same query / key / value weight matrix (W). Q W K W V The sequence slices are subjected to local linear transformations to generate local query matrices (q), local key matrices (k), and local value matrices (v). The dimensions of the local matrices (q, k, v) can be set as follows:

[0078] Where B is the number of videos generated simultaneously, L is the total length of the video token sequence, P is the sequence parallelism (i.e., the number of sequence slices, or the number of GPUs), H is the number of attention heads, and D is the feature vector length of a single attention head.

[0079] In this embodiment, multiple sequence slices are processed by multiple GPUs for local QKV projection calculations to generate corresponding local matrix sets. This parallel processing method using multiple GPUs overcomes the limitations of single-GPU computing resources, enabling parallel processing of long video sequences in a distributed environment and improving the efficiency of video feature extraction and video generation.

[0080] Step S204 The local query matrix, local key matrix, and local value matrix in the multiple sets of local matrices are aggregated to obtain the global query matrix, global key matrix, and global value matrix.

[0081] For the contents of the global query matrix, global key matrix, and global value matrix, please refer to the previous introduction to the local query matrix, local key matrix, and local value matrix.

[0082] In some embodiments, an All-Gather collective communication operation can be used to enable each GPU to obtain all local matrices output by other GPUs, and then aggregate them with its own corresponding local matrices to obtain the corresponding global matrix. This aggregation can be matrix concatenation. In other embodiments, when some GPUs from multiple GPUs are located on the same server, operations such as Reduce-Scatter or Ring-AllGather can be performed via high-speed NVLink to aggregate the local matrices corresponding to these GPUs.

[0083] In this embodiment, aggregation operations are performed on the local query, key, and value matrices on each GPU to obtain the global query, key, and value matrix. This ensures that each GPU can still obtain complete global information in the slice processing mode, providing a data foundation for subsequent self-attention calculations and improving the global consistency of video generation.

[0084] Step S206 Spatiotemporal location encoding is performed on the global query matrix and the global key matrix.

[0085] In some embodiments, the global query matrix and global key matrix can be spatiotemporally encoded on a token-by-token basis using the 3DRoPE method. Specifically, for a token with a temporal-spatial location (t, h, w) in the video token sequence, its spatiotemporal location encoding is achieved through the following complex multiplication:

[0086] Where t represents the position (frame number) of the initial video frame containing the token among multiple initial video frames, h represents the vertical (row) spatial position of the token within the current initial video frame, w represents the horizontal (column) spatial position of the token within the current initial video frame, and θ represents the... T θ H θ W t represents the rotational angular frequencies in the time, height, and width dimensions, respectively. global =t+s represents the global time index of the token (s is the position of the first frame in the initial video frame sequence among multiple initial video frames).

[0087] In other embodiments, the global query matrix and global key matrix can also be spatiotemporally encoded using methods such as absolute position encoding (APE), relative position offset, and decompositional spatiotemporal encoding. During the spatiotemporal encoding process, the encoding computation can be optimized using methods such as trigonometric function pre-computation, GPU-side addressing, and operator fusion.

[0088] In this embodiment, by performing spatiotemporal location encoding on the global query and key matrix, these matrices carry the location information of the video tokens in the temporal and spatial dimensions. This enables subsequent attention calculations to establish spatiotemporal dependencies between tokens based on this location information, thereby improving the consistency and coherence of the generated video in time and space.

[0089] As mentioned earlier, the coding process can be optimized in various ways. The following provides an exemplary optimization method.

[0090] In an optional embodiment, the plurality of initial video frames are multiple adjacent video frames in an initial video frame sequence. For example... Figure 4 As shown, step S206 may include: S400 retrieves the trigonometric function values ​​corresponding to each token from the local cache of each GPU based on the global time index, height information, and width information of each token.

[0091] S402, based on the trigonometric function values ​​corresponding to each token, perform spatiotemporal position encoding on the global query matrix and the global key matrix; wherein, the global time index represents the position of the initial video frame corresponding to the token in the initial video frame sequence.

[0092] For example, if there are 7 video frames in the initial video frame sequence S (numbered 0 to 6), then the multiple initial video frames can be the three video frames numbered 0 to 2.

[0093] In some embodiments, the trigonometric function values ​​in the local cache can be all possible trigonometric function values ​​(including sin and cos) or the complex rotation factors corresponding to the trigonometric function values, pre-calculated based on the maximum sequence length, height, width, and RoPE fundamental frequency θ before video inference. These pre-calculated results can be split into three parts according to dimensions (time, height, and width) (lengths c / / 3, c / / 3, and c-2×(c / / 3), where c represents the feature dimension of a single attention head), and stored in consecutive tensors in the local cache respectively. During the spatiotemporal location encoding process, each GPU can directly index the corresponding sin / cos value (or complex rotation factor) from the pre-calculated consecutive tensors based on the global time index, height information, and width information of each token.

[0094] In spatiotemporal position coding methods such as RoPE, the trigonometric function values ​​corresponding to each token need to be dynamically calculated during the encoding process. Since dynamic calculation relies on host-side (e.g., CPU) operations, it can lead to problems such as synchronization latency and caching logic being unfriendly to overall graph optimization (e.g., discontinuous memory access or fragmented computation stream), causing bottlenecks in video inference performance. Therefore, this embodiment changes the dynamic calculation of trigonometric function values ​​at runtime to pre-calculation and storage in the GPU's local cache. This allows the values ​​to be directly accessed on the GPU during inference without host-side intervention, thus eliminating the synchronization overhead between host and device. Furthermore, pre-calculation combined with contiguous memory (e.g., contiguous tensor) storage can also optimize the memory access patterns of computation streams such as CUDA, reduce computation latency, improve the speed of rotation position coding, and thus improve the overall inference performance and efficiency of video generation.

[0095] Step S208 The global value matrix, the encoded global query matrix, and the encoded global key matrix are divided according to the attention head dimension, and the divided matrices are distributed to the corresponding GPUs so that attention calculations can be performed by the corresponding GPUs to obtain the attention calculation results.

[0096] In some embodiments, after dividing the encoded global key matrix and the encoded global value matrix according to the attention head dimension, the resulting matrices can be stored in a key-value cache (KV Cache) to accumulate historical context information, and then the data in the key-value cache can be reused for calculation during the attention calculation process. Simultaneously, during the attention calculation process, a causal mask can be applied to perform causal attention calculation, ensuring that the current query token can only interact with key tokens at the current and previous positions.

[0097] In this embodiment, the global value matrix, encoded global query matrix, and encoded global key matrix are divided according to the partial attention head that each GPU is responsible for, so that each GPU is responsible for the computational task of a partial attention head. Therefore, after each GPU completes query projection and aggregation of the segmented sequence, it only needs to bear the computational load of a partial attention head, effectively distributing the computational pressure of the self-attention layer in long video generation and improving computational throughput and video generation efficiency.

[0098] In actual processing, the matrix partitioning operation in step S208 can be performed using various methods. An exemplary partitioning method is provided below.

[0099] In an optional embodiment, each GPU corresponds to at least one attention head. For example... Figure 5 As shown, step S208 includes: S500, based on at least one attention head corresponding to each GPU, divide the global value matrix, the encoded global query matrix, and the encoded global key matrix into multiple sets of sub-matrices; each set of sub-matrices corresponds to at least one attention head of the same GPU.

[0100] S502, distribute each sub-matrix in the multiple sub-matrixes to the corresponding GPU.

[0101] For example, if the number of attention heads corresponding to each GPU is set to be the same, then the number of attention heads H can be evenly divided into P groups along the attention head dimension, with each group containing H / P attention heads, where P is the number of GPUs. For the GPU with index r, its corresponding submatrix should contain the global matrix (global value matrix, encoded global query matrix, and encoded global key matrix), and the attention head index starts from... to Feature data.

[0102] In some embodiments, the number of attention heads corresponding to each GPU may also be different. In this case, the actual load of each GPU can be recorded by maintaining a head number offset table, etc.

[0103] In this embodiment, by dividing and redistributing the global query matrix, global key matrix, and global value matrix according to the attention head dimension, each GPU only needs to handle and compute the attention head subset assigned to it. Thus, without compromising the causal consistency of global spatiotemporal location encoding, decoupling and synergizing tensor-dimensional (attention head dimension) parallelism and sequence-dimensional parallelism are achieved. This balances the computational load across multiple GPUs, reduces the peak memory usage of a single GPU, and improves the upper limit of inference length and model scalability for video generation.

[0104] As mentioned earlier, key-value caching can be used to assist in attention calculation, and this will be discussed in detail below.

[0105] In an optional embodiment, each set of submatrices includes a subquery matrix, a subkey matrix, and a subvalue matrix, and each GPU is used to update the key cache and value cache based on the obtained subkey matrix and subvalue matrix.

[0106] In some embodiments, the key cache and value cache are updated by appending to the global timeline sequentially. The key cache and value cache only store the key-value pairs corresponding to the multiple initial video frames currently being processed, as well as the video frames preceding the multiple initial video frames currently being processed in the initial video frame sequence.

[0107] For example, if there are 7 video frames in the initial video frame sequence S (numbered 0 to 6), and the current processing of multiple initial video frames are two of them with numbers 3 and 4, then the GPU will append the subkey matrix and subvalue matrix corresponding to the two video frames with numbers 3 and 4 to the key cache and value cache. After the update, the key cache and value cache will store the subkey matrix and subvalue matrix corresponding to the video frames with numbers 0 to 4.

[0108] In other embodiments, the key cache and value cache can also be incrementally updated using a rolling update method. When the data volume reaches the upper limit, the oldest cached data (sub-value matrix or sub-key matrix) can be automatically eliminated.

[0109] In this embodiment, by maintaining key and value caches on each GPU and performing incremental updates based on subkey and subvalue matrices, the attention calculation results of historical frames can be reused during the video block autoregressive generation process. This avoids repeatedly encoding historical global information for each newly generated block, reducing computational redundancy and peak memory usage in long video generation scenarios. At the same time, it maintains the consistency of causal attention during the video generation process, improving the computational efficiency and video quality of video generation.

[0110] The updated key-value cache can be applied to specific attention calculation processes. The following describes how to use the updated key-value cache for attention calculation.

[0111] In an optional embodiment, step S208 includes: The attention calculation results of the corresponding attention head are obtained by performing attention calculations on the corresponding GPU based on its updated key cache and value cache, as well as the corresponding subquery matrix.

[0112] In some embodiments, each GPU can perform scaled dot product attention computation in parallel on its assigned subset of attention heads, using the following formula:

[0113] Among them, Q h K h V h These are the subquery matrix, subkey matrix, and subvalue matrix corresponding to the h-th attention head on this GPU, respectively, d k Let be the dimension of the key vector. During the above operations, a block-based strategy can be used to adapt to the GPU's shared memory capacity. Through cyclic block generation and register-level data reuse, the number of accesses to global memory can be reduced. In some embodiments, matrix multiplication scheduling can also employ a row-column interleaved memory layout to optimize kernel access patterns.

[0114] In this embodiment, attention calculation is performed using the GPU's local key-value cache, enabling each GPU to reuse cached historical key-value pairs when performing attention calculations, without having to repeatedly calculate previously generated video frames. This reduces the computational complexity and overhead of video generation, and improves the computational efficiency of video generation.

[0115] Step S210 It aggregates the attention calculation results from each GPU.

[0116] In some embodiments, attention calculation results from local attention heads assigned to different GPUs can be aggregated into a complete attention calculation result based on the attention head dimension. Thus, each GPU holds the attention calculation results from all attention heads. In some embodiments, before aggregating the attention calculation results from each GPU, causal mask verification can be applied to each local attention result to ensure that in the current block autoregressive generation step, each GPU only aggregates the valid attention calculation results corresponding to the generated block. Alternatively, the attention weights calculated by each GPU can be checked for consistency with the corresponding global key-value pairs to ensure they match the attention head dimension assigned to each GPU in step S208.

[0117] Step S212 Multiple target video frames are generated based on the aggregated attention calculation results, corresponding to the multiple initial video frames.

[0118] In some embodiments, the aggregated attention calculation results can be redistributed to the GPUs according to the sequence dimension, thereby enabling each GPU to receive the attention calculation results of all attention heads for the sequence slice it processes. The specific allocation and calculation process will be described in detail later. In other embodiments, the aggregation and redistribution of the attention calculation results of multiple GPUs can also be implemented synchronously through operations such as All-To-All. That is, each GPU can output its local attention calculation results according to the attention head dimension, while simultaneously aggregating the attention calculation results from other GPUs according to the sequence dimension, thus converting the layout of the attention calculation results from a tensor-parallel layout to a sequence-parallel layout. After obtaining the attention calculation results of all attention heads for the sequence slice it processes, each GPU can generate the target video frame by performing cross-attention calculations with pre-input text features and repeating the above-described denoising process.

[0119] The following describes an exemplary implementation of step S212.

[0120] In optional embodiments, such as Figure 6 As shown, step S212 includes: S600 divides the aggregated attention calculation results into multiple segmented attention calculation results by each GPU; one segmented attention calculation result corresponds to a sequence slice processed by one GPU.

[0121] S602 generates video results by performing cross-attention calculations on each GPU based on a pre-set text description and the attention calculation results after segmentation corresponding to that GPU, thereby generating the video segments corresponding to that GPU.

[0122] S604, aggregate the video generation results of the sequence segments corresponding to each GPU to obtain the multiple target video frames.

[0123] In some embodiments, a pre-set text description can be processed by a text encoder to obtain a corresponding semantic token. After performing cross-attention calculation using the obtained semantic token and the corresponding attention calculation result, the result of the cross-attention calculation is a token sequence with the same length as the sequence slice input in step S202. In some embodiments, the result of the cross-attention calculation can also be used as input to re-execute the operations from steps S202 to S212 to achieve multiple denoising iterations and finally obtain the video generation result of the sequence slice corresponding to the GPU. The video generation result obtained by each GPU can be a latent representation of the video, which can be restored to pixel space by the decoder of a 3D causal variational autoencoder to obtain the target video frame.

[0124] In this embodiment, by redistributing the attention calculation results to each GPU according to the sequence fragment dimension, and independently executing cross-attention calculations related to text description on each GPU, a combination of distributed parallel generation and text control is achieved. This avoids the memory bottleneck of a single GPU processing the entire video sequence while maintaining the fine-grained alignment capability from text to video generation. Simultaneously, through a fragment-level result aggregation strategy, the data consistency problem in multi-GPU collaborative generation under a sequence parallel architecture is solved. In scenarios supporting long video block autoregressive generation, this reduces cross-GPU communication overhead and improves overall inference throughput, thereby enhancing the computational efficiency and scalability of video generation.

[0125] To make this embodiment easier to understand, the following is combined with... Figure 7 An example application is provided. Wherein: Step 1: Assume that the video to be generated is encoded by 3D Causal VAE to form a video token sequence with a grid size of (F=4, H=4, W=4) and a total sequence length of L=64. Configure the batch size B=1 and the sequence parallelism P=2. Divide the sequence into two pieces of length 32 and input them into GPU0 and GPU1 respectively. In each GPU, a local query matrix q, a local key matrix k, and a local value matrix v with shape (1, 32, H, D) are generated through a local QKV projection layer. Here, F, H, and W represent the number of frames (time dimension), spatial height, and spatial width (the unit is the number of tokens).

[0126] Step 2: Perform AllGather communication between the two GPUs to aggregate local fragments along the sequence dimension (dim=1), so that each GPU obtains a complete global query matrix, global key matrix, and global value matrix, all with the shape (1, 64, H, D), where H is the number of attention heads and D is the dimension of the attention heads.

[0127] Step 3: Set the global starting frame offset s=0 for the current generated block. Based on the grid size (F, H, W) and the RoPE frequency θ in the GPU local cache, calculate the global position index of each token in three-dimensional spacetime. Apply 3D rotation position encoding to the global query matrix and the global key matrix, introduce global time information, and obtain the encoded matrix.

[0128] Step 4: Divide the encoded global key matrix and global value matrix along the attention head dimension (dim=2), and assign the first H / 2 attention heads to GPU0 and the last H / 2 to GPU1 to perform parallel computation along the attention head dimension.

[0129] Step 5: Store the key matrix fragments and value matrix fragments allocated to each GPU after splitting into the KV cache of the corresponding GPU for reuse of historical information in the current and subsequent generation steps.

[0130] Step 6: Each GPU performs scaled dot product attention calculation based on the key-value pairs in its local KV cache and the global query matrix of the corresponding attention head slice, to obtain a local attention output with shape (1, 64, H / 2, D).

[0131] Step 7: Perform an All-to-all communication operation, aggregating the local outputs of each GPU along the attention head dimension and redistributing them along the sequence dimension, so that GPU0 recovers the full-head attention output corresponding to the first 32 token positions, and GPU1 obtains the output corresponding to the last 32 token positions. The shape is restored to (1, 32, H, D). After concatenation, the sequence parallel self-attention calculation in the current denoising step is completed. After the output result is cross-attention calculated with the text, it can be decoded into the target video frame after multiple denoising steps.

[0132] Example 2 Figure 8 A flowchart illustrating a video generation method according to Embodiment 2 of this application is shown schematically.

[0133] like Figure 8 As shown, the video generation method may include steps S800~S812, wherein: S800, the video token sequence is divided into multiple sequence segments. The video token sequence is formed based on multiple initial video frames to be processed. The multiple initial video frames are multiple adjacent video frames in the initial video frame sequence.

[0134] S802, the multiple sequence slices are distributed to multiple GPUs for processing to obtain multiple sets of local matrices. Each GPU corresponds to one sequence slice and one set of local matrices. One set of local matrices includes a local query matrix, a local key matrix, and a local value matrix.

[0135] S804, in each GPU, based on the position of the first frame in the initial video frame sequence, spatiotemporal position encoding is performed on the respective local query matrix and local key matrix.

[0136] S806 performs cross-GPU data redistribution across multiple GPUs, on their respective local value matrices, encoded local query matrices, and encoded local key matrices, so that each GPU obtains a matrix to be computed for attention computation.

[0137] S808 uses each GPU to perform attention calculations based on the obtained matrix to be calculated in order to obtain the attention calculation result.

[0138] S810 aggregates the attention calculation results from each GPU.

[0139] S812, generate multiple target video frames corresponding to the multiple initial video frames based on the aggregated attention calculation results.

[0140] The video generation method provided in this embodiment distributes video sequence fragments to multiple GPUs for parallel processing, and independently completes spatiotemporal position encoding on each GPU by combining the global offset of the first frame. Then, it uses fused data redistribution to replace multiple global communications, which simplifies the computation and communication process, reduces the memory pressure and communication latency of long video generation, and improves the computational efficiency and real-time generation performance of video generation.

[0141] Some technical details in this embodiment can be found in Embodiment 1, such as S800, S802, S808, S810, and S812.

[0142] The following section details several differences between the video generation method in this embodiment and the video generation method in Embodiment 1.

[0143] Difference 1: In an optional embodiment, each sequence shard includes multiple tokens, such as Figure 9 As shown, step S804 includes: S900, based on the position of the first frame in the initial video frame sequence, determine the global time index of each token.

[0144] S902 retrieves the trigonometric function values ​​corresponding to each token from the local cache of each GPU based on the global time index, height information, and width information of each token.

[0145] S904, based on the trigonometric function values ​​corresponding to each token, perform spatiotemporal position encoding on the local query matrix and the local key matrix; wherein, the global time index represents the position of the initial video frame corresponding to the token in the initial video frame sequence.

[0146] The global time index t for each token global It can be calculated using the following formula:

[0147] Where s is the position of the first frame in the initial video frame sequence, H and W represent the spatial height and spatial width of each initial video frame, respectively, and i global The position index of the token within the sequence fragment (the tokens are arranged in frame priority order within the sequence fragment) can be obtained using the following formula:

[0148] Where r represents the sequence number of the sequence segment obtained from the video feature block in this processing, L is the number of tokens in each sequence segment, and i local This indicates the position of the token within the sequence fragment.

[0149] During the spatiotemporal location encoding process, low-level tools (such as TileLang) can be used to implement operator fusion, integrating operations such as obtaining trigonometric function values ​​and complex multiplication / rotation calculations into the same kernel function. In some embodiments, the token can also be converted to complex form (torch.view_as_complex) and directly multiplied element-wise with the complex form of rotation frequencies (freqs).

[0150] Since rotational position encoding requires complete sequence context information, it is necessary to synchronize the complete matrix corresponding to the global token sequence through three cross-GPU communications before position encoding calculation can be performed. This results in strong coupling between the computational and communication logics, and the dense communication overhead limits the speed of video inference generation. Therefore, this embodiment introduces the position of the first frame in the initial video frame sequence from multiple initial video frames, transforming the position encoding, which originally relied on global information, into an equivalent calculation that relies only on local features and global temporal indexes. This allows each GPU to independently complete the position encoding of its assigned slice without cross-node communication, improving the efficiency and speed of video inference generation.

[0151] Difference 2: In an optional embodiment, step S806 includes: In each GPU, the local value matrix, the encoded local query matrix, and the encoded local key matrix are divided into multiple data slices according to the attention head dimension and distributed to the corresponding other GPUs. The data slices received from other GPUs are then concatenated according to the sequence dimension so that each GPU obtains a matrix to be computed for attention calculation.

[0152] For example, two GPUs (GPU0 and GPU1) jointly process a video sequence containing 100 tokens and 2 attention heads: GPU0 holds all the header information of the first 50 tokens, it splits the second header data and sends it to GPU1, and at the same time receives the data of the first header from the last 50 tokens sent by GPU1, thus splicing them locally on GPU0 into a complete matrix containing all 100 tokens but only for the first header, realizing the data recombination from "segmentation by sequence dimension" to "segmentation by attention head dimension".

[0153] In some embodiments, the local query matrix, the encoded local query matrix, and the encoded local key matrix can be concatenated or merged in memory before distribution to form a continuous communication buffer.

[0154] Based on the local spatiotemporal location encoding performed by each GPU in step S804, this embodiment achieves the replacement of multiple aggregation communication and segmentation operations by simultaneously completing the transformation and reconstruction of the local query matrix, local key matrix, and local value matrix in the sequence dimension and attention head dimension in a single communication step. This reduces the performance loss caused by cross-GPU communication and improves the inference speed and efficiency of video generation.

[0155] To make this embodiment easier to understand, the following is combined with... Figure 10 An example application is provided. Wherein: Step 1: Divide the video token sequence corresponding to the video to be generated into 4 sequence fragments according to the sequence parallelism, and distribute them to GPU0 to GPU3. Each GPU independently performs QKV linear projection on its local fragment to obtain the local query, key, and value matrix.

[0156] Step 2, calculate the global token offset within each GPU: After determining the local sequence length, calculate the starting position offset of the current GPU in the global sequence, so that each GPU knows the global sequence range it is responsible for.

[0157] Step 3: Based on the starting frame offset 6 of the current generated block in the global video sequence and the global token offset of each GPU, calculate the global time index of each local token, index the corresponding time, height and width three-dimensional frequencies from the local pre-cached rotation frequency parameters, and perform causal three-dimensional rotation position encoding on the query and key matrix. This calculation is completed independently on each GPU without the need for cross-device communication.

[0158] Step 4: Perform fused All-to-All communication to convert the query, key, and value tensors of each GPU from a sequence-parallel layout to an attention-head-parallel layout, complete data aggregation in the sequence dimension, and split it in the attention-head dimension.

[0159] Step 5: Update the transformed key matrix and value matrix to the KV cache. The stored key-value pairs carry global time position information to ensure time sequence consistency.

[0160] Step 6: Each GPU reads key-value pairs containing historical information from the KV cache and performs causal attention calculation with the current query matrix. The query only interacts with keys in the cache whose time index is earlier than or equal to the current time, generating attention outputs for each GPU.

[0161] Step 7: Perform All-to-All communication to restore the attention output from tensor parallel layout to sequence parallel layout, aggregate the attention output results of each GPU to obtain the complete attention output of the current denoising step, and generate the video frame corresponding to the current block through subsequent processing such as cross-attention calculation with text. Repeat the iteration for 40 to 50 steps of denoising until the target video is generated.

[0162] Example 3 Figure 11 The diagram schematically illustrates a video generation apparatus according to Embodiment 3 of this application. This apparatus can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program modules referred to in the embodiments of this application are a series of computer program instruction segments capable of performing specific functions. The following description will specifically introduce the functions of each program module in this embodiment. For example... Figure 11 As shown, the device 1000 may include: a fragmentation module 1100, a processing module 1200, a first aggregation module 1300, an encoding module 1400, a distribution module 1500, a second aggregation module 1600, and a generation module 1700, wherein: The segmentation module 1100 is used to divide the video token sequence into multiple sequence segments, and the video token sequence is formed based on multiple initial video frames to be processed. Processing module 1200 is used to distribute the multiple sequence slices to multiple GPUs for processing, so as to obtain multiple sets of local matrices. One GPU corresponds to one sequence slice and one set of local matrices. One set of local matrices includes a local query matrix, a local key matrix and a local value matrix. The first aggregation module 1300 is used to aggregate the local query matrix, local key matrix and local value matrix in the multiple sets of local matrices respectively to obtain the global query matrix, global key matrix and global value matrix. Encoding module 1400 is used to perform spatiotemporal position encoding on the global query matrix and the global key matrix; The distribution module 1500 is used to divide the global value matrix, the encoded global query matrix, and the encoded global key matrix according to the attention head dimension, and distribute the divided matrices to the corresponding GPUs so that attention calculations can be performed by the corresponding GPUs to obtain the attention calculation results. The second aggregation module 1600 is used to aggregate the attention calculation results of each GPU; and The generation module 1700 is used to generate multiple target video frames corresponding to the multiple initial video frames based on the aggregated attention calculation results.

[0163] As an optional embodiment, the video token sequence includes multiple tokens, and the segmentation module is further used for: The number of tokens in each sequence slice is determined based on the number of multiple sequence slices and the number of multiple tokens; The video token sequence is divided into multiple sequence segments based on the number of tokens in each sequence segment.

[0164] As an optional embodiment, the plurality of initial video frames are multiple adjacent video frames in an initial video frame sequence, and the encoding module is further configured to: Based on the global time index, height information, and width information of each token, retrieve the trigonometric function values ​​corresponding to each token from the local cache of each GPU; and Based on the trigonometric function values ​​corresponding to each token, the global query matrix and global key matrix are spatiotemporally encoded. The global time index represents the position of the initial video frame corresponding to the token in the initial video frame sequence.

[0165] As an optional embodiment, each GPU corresponds to at least one attention head, and the distribution module is also used for: Based on at least one attention head corresponding to each GPU, the global value matrix, the encoded global query matrix, and the encoded global key matrix are each divided into multiple sub-matrices; each sub-matrix corresponds to at least one attention head of the same GPU; and Distribute each of the multiple sets of sub-matrices to the corresponding GPU.

[0166] As an optional embodiment, each set of submatrices includes a subquery matrix, a subkey matrix, and a subvalue matrix, and each GPU is used to update the key cache and value cache based on the obtained subkey matrix and subvalue matrix.

[0167] As an optional embodiment, the device 1000 further includes a computing module for: The attention calculation results of the corresponding attention head are obtained by performing attention calculations on the corresponding GPU based on its updated key cache and value cache, as well as the corresponding subquery matrix.

[0168] As an optional embodiment, the generation module is also used for: The aggregated attention calculation results are segmented by each GPU to obtain multiple segmented attention calculation results; each segmented attention calculation result corresponds to a sequence slice processed by one GPU. Each GPU performs cross-attention calculations based on a pre-set text description and the attention calculation results of the segmented data corresponding to that GPU, to generate the video generation result of the sequence segments corresponding to that GPU; and The video generation results of the sequence segments corresponding to each GPU are aggregated to obtain the multiple target video frames.

[0169] Example 4 Figure 12 The diagram schematically illustrates a video generation apparatus according to Embodiment 4 of this application. This apparatus can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program modules referred to in the embodiments of this application are a series of computer program instruction segments capable of performing specific functions. The following description will specifically introduce the functions of each program module in this embodiment. For example... Figure 12 As shown, the device 2000 may include: a fragmentation module 2100, a processing module 2200, an encoding module 2300, a redistribution module 2400, a calculation module 2500, an aggregation module 2600, and a generation module 2700, wherein: The segmentation module 2100 is used to divide the video token sequence into multiple sequence segments. The video token sequence is formed based on multiple initial video frames to be processed. The multiple initial video frames are multiple adjacent video frames in the initial video frame sequence. Processing module 2200 is used to distribute the multiple sequence slices to multiple GPUs for processing to obtain multiple sets of local matrices. One GPU corresponds to one sequence slice and one set of local matrices. One set of local matrices includes a local query matrix, a local key matrix, and a local value matrix. Encoding module 2300 is used to perform spatiotemporal position encoding on the local query matrix and local key matrix of each GPU based on the position of the first frame in the initial video frame sequence among the plurality of initial video frames; The redistribution module 2400 is used to perform cross-GPU data redistribution on the local value matrix, the encoded local query matrix, and the encoded local key matrix of each GPU, so that each GPU can obtain the matrix to be computed for attention calculation. The computing module 2500 is used to perform attention calculations on each GPU based on the obtained matrix to be calculated in order to obtain the attention calculation results. The aggregation module 2600 is used to aggregate the attention calculation results from each GPU; and The generation module 2700 is used to generate multiple target video frames corresponding to the multiple initial video frames based on the aggregated attention calculation results.

[0170] As an optional embodiment, each sequence fragment includes multiple tokens, and the encoding module 2300 is further used for: Based on the position of the first frame in the initial video frame sequence, the global time index of each token is determined. Based on the global time index, height information, and width information of each token, retrieve the trigonometric function values ​​corresponding to each token from the local cache of each GPU; Based on the trigonometric function values ​​corresponding to each token, the local query matrix and local key matrix are spatiotemporally encoded. The global time index represents the position of the initial video frame corresponding to the token in the initial video frame sequence.

[0171] As an optional embodiment, the redistribution module 2400 is also used for: In each GPU, the local value matrix, the encoded local query matrix, and the encoded local key matrix are divided into multiple data slices according to the attention head dimension and distributed to the corresponding other GPUs. The data slices received from other GPUs are then concatenated according to the sequence dimension so that each GPU obtains a matrix to be computed for attention calculation.

[0172] Example 5 Figure 13 This illustration schematically shows a hardware architecture diagram of a computer device 10000 suitable for implementing a video generation method according to Embodiment 5 of this application. In some embodiments, the computer device 10000 may be a workstation, industrial terminal, personal computer, vehicle terminal, game console, virtual device, robot, or other terminal device. In other embodiments, the computer device 10000 may be a rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers), etc. Figure 13 As shown, the computer device 10000 includes, but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate and be linked with each other via a system bus. Wherein: The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of a computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as program code for video generation methods. In addition, the memory 10010 can also be used to temporarily store various types of data that have been output or will be output.

[0173] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.

[0174] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 10000 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0175] It should be pointed out that, Figure 13 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0176] In this embodiment, the video generation method stored in memory 10010 can be further divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of this application.

[0177] Example 6 This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the video generation method in the embodiments.

[0178] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the video generation method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0179] Example 7 This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in the above embodiments.

[0180] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computer devices. They can be centralized on a single computer device or distributed across a network of multiple computer devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computer device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0181] It should be noted that the above are merely preferred embodiments of this application and do not limit the scope of patent protection of this application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A video generation method, characterized in that, The method includes: The video token sequence is divided into multiple sequence segments, and the video token sequence is formed based on multiple initial video frames to be processed. The multiple sequence slices are distributed to multiple GPUs for processing to obtain multiple sets of local matrices. Each GPU corresponds to one sequence slice and one set of local matrices. A set of local matrices includes a local query matrix, a local key matrix, and a local value matrix. The local query matrix, local key matrix, and local value matrix in the multiple sets of local matrices are aggregated to obtain the global query matrix, global key matrix, and global value matrix. Spatiotemporal location encoding is performed on the global query matrix and the global key matrix; The global value matrix, the encoded global query matrix, and the encoded global key matrix are divided according to the attention head dimension, and the divided matrices are distributed to the corresponding GPUs so that attention calculations can be performed by the corresponding GPUs to obtain the attention calculation results. Aggregate the attention calculation results from each GPU; and Multiple target video frames are generated based on the aggregated attention calculation results, corresponding to the multiple initial video frames.

2. The method according to claim 1, characterized in that, The video token sequence includes multiple tokens; The video token sequence is divided into multiple sequence fragments, including: The number of tokens in each sequence slice is determined based on the number of multiple sequence slices and the number of multiple tokens; The video token sequence is divided into multiple sequence segments based on the number of tokens in each sequence segment.

3. The method according to claim 2, characterized in that, The plurality of initial video frames are multiple adjacent video frames in the initial video frame sequence; Spatiotemporal location encoding of the global query matrix and global key matrix includes: Based on the global time index, height information, and width information of each token, retrieve the trigonometric function values ​​corresponding to each token from the local cache of each GPU; and Based on the trigonometric function values ​​corresponding to each token, the global query matrix and global key matrix are spatiotemporally encoded. The global time index represents the position of the initial video frame corresponding to the token in the initial video frame sequence.

4. The method according to claim 1, characterized in that, Each GPU corresponds to at least one attention head; The global value matrix, encoded global query matrix, and encoded global key matrix are divided according to the attention head dimension, and the divided matrices are distributed to the corresponding GPUs, including: Based on at least one attention head corresponding to each GPU, the global value matrix, the encoded global query matrix, and the encoded global key matrix are each divided into multiple sub-matrices; each sub-matrix corresponds to at least one attention head of the same GPU; and Distribute each of the multiple sets of sub-matrices to the corresponding GPU.

5. The method according to claim 4, characterized in that, Each submatrix includes a subquery matrix, a subkey matrix, and a subvalue matrix. Each GPU is used to update the key cache and value cache based on the obtained subkey matrix and subvalue matrix.

6. The method according to claim 5, characterized in that, Attention calculations are performed using the corresponding GPU to obtain the attention calculation results, including: The attention calculation results of the corresponding attention head are obtained by performing attention calculations on the corresponding GPU based on its updated key cache and value cache, as well as the corresponding subquery matrix.

7. The method according to claim 1, characterized in that, Based on the aggregated attention calculation results, multiple target video frames corresponding to the multiple initial video frames are generated, including: The aggregated attention calculation results are segmented by each GPU to obtain multiple segmented attention calculation results; each segmented attention calculation result corresponds to a sequence slice processed by one GPU. Each GPU performs cross-attention calculations based on a pre-set text description and the attention calculation results of the segmented data corresponding to that GPU, to generate the video generation result of the sequence segments corresponding to that GPU; and The video generation results of the sequence segments corresponding to each GPU are aggregated to obtain the multiple target video frames.

8. A video generation method, characterized in that, The method includes: The video token sequence is divided into multiple sequence segments. The video token sequence is formed based on multiple initial video frames to be processed. The multiple initial video frames are multiple adjacent video frames in the initial video frame sequence. The multiple sequence slices are distributed to multiple GPUs for processing to obtain multiple sets of local matrices. Each GPU corresponds to one sequence slice and one set of local matrices. A set of local matrices includes a local query matrix, a local key matrix, and a local value matrix. In each GPU, spatiotemporal position encoding is performed on the local query matrix and local key matrix based on the position of the first frame in the initial video frame sequence. In multiple GPUs, cross-GPU data redistribution is performed on the local value matrix, the encoded local query matrix, and the encoded local key matrix, so that each GPU obtains the matrix to be computed for attention computation. The attention calculation result is obtained by performing attention calculation on each GPU based on the obtained matrix to be calculated. Aggregate the attention calculation results from each GPU; and Multiple target video frames are generated based on the aggregated attention calculation results, corresponding to the multiple initial video frames.

9. The method according to claim 8, characterized in that, Each sequence shard includes multiple tokens; Based on the position of the first frame in the initial video frame sequence, spatiotemporal position encoding is performed on their respective local query matrices and local key matrices, including: Based on the position of the first frame in the initial video frame sequence, the global time index of each token is determined. Based on the global time index, height information, and width information of each token, retrieve the trigonometric function values ​​corresponding to each token from the local cache of each GPU; Based on the trigonometric function values ​​corresponding to each token, the local query matrix and local key matrix are spatiotemporally encoded. The global time index represents the position of the initial video frame corresponding to the token in the initial video frame sequence.

10. The method according to claim 8, characterized in that, Across multiple GPUs, cross-GPU data redistribution is performed on the respective local value matrices, encoded local query matrices, and encoded local key matrices, including: In each GPU, the local value matrix, the encoded local query matrix, and the encoded local key matrix are divided into multiple data slices according to the attention head dimension and distributed to the corresponding other GPUs. The data slices received from other GPUs are then concatenated according to the sequence dimension so that each GPU obtains a matrix to be computed for attention calculation.

11. A video generation apparatus, characterized in that, The device includes: The segmentation module is used to divide the video token sequence into multiple sequence segments. The video token sequence is formed based on multiple initial video frames to be processed. The processing module is used to distribute the multiple sequence slices to multiple GPUs for processing, so as to obtain multiple sets of local matrices. One GPU corresponds to one sequence slice and one set of local matrices. One set of local matrices includes a local query matrix, a local key matrix and a local value matrix. The first aggregation module is used to aggregate the local query matrix, local key matrix and local value matrix in the multiple sets of local matrices respectively to obtain the global query matrix, global key matrix and global value matrix; The encoding module is used to perform spatiotemporal position encoding on the global query matrix and the global key matrix; The distribution module is used to divide the global value matrix, the encoded global query matrix, and the encoded global key matrix according to the attention head dimension, and distribute the divided matrices to the corresponding GPUs so that attention calculations can be performed by the corresponding GPUs to obtain the attention calculation results. The second aggregation module is used to aggregate the attention calculation results of each GPU; and The generation module is used to generate multiple target video frames corresponding to the multiple initial video frames based on the aggregated attention calculation results.

12. A video generation apparatus, characterized in that, The device includes: The segmentation module is used to divide the video token sequence into multiple sequence segments. The video token sequence is formed based on multiple initial video frames to be processed. The multiple initial video frames are multiple adjacent video frames in the initial video frame sequence. The processing module is used to distribute the multiple sequence slices to multiple GPUs for processing, so as to obtain multiple sets of local matrices. One GPU corresponds to one sequence slice and one set of local matrices. One set of local matrices includes a local query matrix, a local key matrix and a local value matrix. The encoding module is used in each GPU to perform spatiotemporal position encoding on the local query matrix and local key matrix based on the position of the first frame in the initial video frame sequence among the plurality of initial video frames. The redistribution module is used to perform cross-GPU data redistribution on the local value matrix, the encoded local query matrix, and the encoded local key matrix of each GPU, so that each GPU can obtain the matrix to be computed for attention calculation. The computation module is used to perform attention calculations on each GPU based on the obtained matrix to be calculated in order to obtain the attention calculation results; The aggregation module is used to aggregate the attention calculation results from each GPU; and The generation module is used to generate multiple target video frames corresponding to the multiple initial video frames based on the aggregated attention calculation results.

13. A computer device, characterized in that, include: At least one processor; and A memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 10.

15. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 10.

Images