Video generation method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to a video generation method and device, electronic equipment and storage medium. The method comprises: obtaining a text description; inputting the text description into a pre-trained text-to-video model to generate a target video matching the text description; the text-to-video model comprises a feature mapping module, an attention module and a video generation module; the feature mapping module is configured to convert the text description into a video feature sequence, and map the video feature sequence into a query feature matrix, a key feature matrix and a value feature matrix respectively; the attention module is configured to perform a preset attention operation on the query feature matrix and the key feature matrix to obtain a first matrix and a second matrix, obtain a target attention matrix based on the first matrix and the second matrix, and obtain a target attention output based on the target attention matrix and the value feature matrix; and the video generation module is configured to generate the target video based on the target attention output. The present disclosure can improve the video generation efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to video generation methods, apparatus, electronic devices, and storage media. Background Technology

[0002] Text-based video technology can generate video content that users need based on the text descriptions they input.

[0003] With the continuous development of text-based video technology, traditional text-based video technology can use the diffusion model based on the Transformer architecture to generate video content that matches the text description input by the user. However, the attention mechanism used by the diffusion model based on the Transformer architecture in traditional technology often has high computational and memory complexity. When reasoning about video content with a large number of frames, the operation speed is slow and the computation cost is high, resulting in low video generation efficiency. Summary of the Invention

[0004] This disclosure provides a video generation method, apparatus, electronic device, computer-readable storage medium, and computer program product to at least solve the problem of low video generation efficiency in related video generation technologies. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a video generation method is provided, comprising:

[0006] Get the text description;

[0007] The text description is input into a pre-trained text-based video model to generate a target video that matches the text description; the text-based video model includes a feature mapping module, an attention module, and a video generation module, wherein;

[0008] The feature mapping module is used to convert the text description into a video feature sequence, and map the video feature sequence into a query feature matrix, a key feature matrix and a value feature matrix, respectively;

[0009] The attention module is used to perform a preset attention operation on the query feature matrix and the key feature matrix to obtain a first matrix and a second matrix; the first matrix is ​​used to obtain the correlation information between the same spatiotemporal blocks of different video frames in the video feature sequence; the second matrix is ​​used to obtain the correlation information between different spatiotemporal blocks of the same video frame in the video feature sequence.

[0010] The attention module is further configured to obtain a target attention matrix based on the first matrix and the second matrix, and to obtain a target attention output based on the target attention matrix and the value feature matrix;

[0011] The video generation module is used to generate the target video based on the target attention output.

[0012] In one possible implementation, obtaining the target attention matrix based on the first matrix and the second matrix includes:

[0013] Obtain the number of video frames and the number of spatiotemporal blocks per frame in the video feature sequence;

[0014] The permutation matrix is ​​determined based on the number of video frames and the number of spatiotemporal blocks per frame; the permutation matrix includes a first permutation matrix associated with the first matrix and a second permutation matrix associated with the second matrix;

[0015] The target attention matrix is ​​generated based on the permutation matrix, the first matrix, and the second matrix;

[0016] The target attention matrix is ​​the product of the first permutation matrix, the first matrix, the second permutation matrix, and the second matrix.

[0017] In one possible implementation, performing a preset attention operation on the query feature matrix and the key feature matrix to obtain a first matrix and a second matrix includes:

[0018] Initialize the first matrix and the second matrix;

[0019] Fix the first matrix, and update the second matrix based on the first matrix and the key feature matrix;

[0020] The updated second matrix is ​​fixed, and the first matrix is ​​updated based on the updated second matrix and the query feature matrix;

[0021] The steps of returning to the fixed first matrix and updating the second matrix based on the first matrix and the key feature matrix continue until the difference between the target attention matrix and the full attention matrix meets a preset condition; the full attention matrix is ​​the attention matrix obtained by performing full attention operation on the query feature matrix and the key feature matrix.

[0022] In one possible implementation, updating the second matrix based on the first matrix and the key feature matrix includes:

[0023] Obtain the product of the first matrix and the query feature matrix to get the first intermediate matrix when updating the second matrix;

[0024] Based on the first matrix, determine the columns and parameter items for updating the second matrix;

[0025] The second matrix is ​​updated based on the key feature matrix, the columns and parameter items, and the first intermediate matrix.

[0026] In one possible implementation, updating the first matrix based on the updated second matrix and the query feature matrix includes:

[0027] Obtain the product of the second matrix and the key feature matrix to obtain the second intermediate matrix when updating the first matrix;

[0028] Based on the second matrix, determine the rows and parameter items for updating the first matrix;

[0029] The first matrix is ​​updated based on the query feature matrix, the rows and parameter items, and the second intermediate matrix.

[0030] In one possible implementation, each video frame of the video feature sequence includes a first preset number of spatiotemporal blocks, and the first matrix includes submatrices corresponding to each of the spatiotemporal blocks.

[0031] Wherein, the element value of the kth submatrix of the first matrix represents the temporal correlation weight between the kth spatiotemporal block in the i-th video frame and the kth spatiotemporal block in the j-th video frame of the video feature sequence.

[0032] Where k, i, and j are all positive integers.

[0033] In one possible implementation, the video feature sequence includes a second preset number of video frames, and the second matrix includes a submatrix corresponding to each of the video frames.

[0034] Wherein, the element value of the kth submatrix of the second matrix represents the spatial correlation weight between the i-th spatiotemporal block in the k-th video frame of the video feature sequence and the j-th spatiotemporal block in the k-th video frame;

[0035] Where k, i, and j are all positive integers.

[0036] In one possible implementation, the method further includes:

[0037] Obtain the query feature matrix corresponding to the preset frame video frame, and use it as the preset frame query feature matrix;

[0038] Perform full attention operation on the preset frame query feature matrix and the key feature matrix to obtain the preset frame attention matrix, and obtain the preset frame attention output based on the preset frame attention matrix and the value feature matrix;

[0039] The preset frame attention output is used to cover a preset portion of the target attention output to obtain the optimized target attention output; wherein, the preset portion is the attention output corresponding to the preset frame video frame in the target attention output.

[0040] According to a second aspect of the present disclosure, a video generation apparatus is provided, comprising:

[0041] The retrieval unit is configured to retrieve the text description.

[0042] The generation unit is configured to input the text description into a pre-trained text-based video model and generate a target video that matches the text description; the text-based video model includes a feature mapping module, an attention module, and a video generation module, wherein;

[0043] The feature mapping module is used to convert the text description into a video feature sequence, and map the video feature sequence into a query feature matrix, a key feature matrix and a value feature matrix, respectively;

[0044] The attention module is used to perform a preset attention operation on the query feature matrix and the key feature matrix to obtain a first matrix and a second matrix; the first matrix is ​​used to obtain the correlation information between the same spatiotemporal blocks of different video frames in the video feature sequence; the second matrix is ​​used to obtain the correlation information between different spatiotemporal blocks of the same video frame in the video feature sequence.

[0045] The attention module is further configured to obtain a target attention matrix based on the first matrix and the second matrix, and to obtain a target attention output based on the target attention matrix and the value feature matrix;

[0046] The video generation module is used to generate the target video based on the target attention output.

[0047] In one possible implementation, the generation unit is specifically configured to perform the following actions: acquiring the number of video frames and the number of spatiotemporal blocks per frame of the video feature sequence; determining a permutation matrix based on the number of video frames and the number of spatiotemporal blocks per frame; the permutation matrix including a first permutation matrix associated with the first matrix and a second permutation matrix associated with the second matrix; and generating the target attention matrix based on the permutation matrix, the first matrix, and the second matrix; wherein the target attention matrix is ​​the product of the first permutation matrix, the first matrix, the second permutation matrix, and the second matrix.

[0048] In one possible implementation, the generation unit is specifically configured to perform the following steps: initializing the first matrix and the second matrix; fixing the first matrix and updating the second matrix based on the first matrix and the key feature matrix; fixing the updated second matrix and updating the first matrix based on the updated second matrix and the query feature matrix; returning to the steps of fixing the first matrix and updating the second matrix based on the first matrix and the key feature matrix, until the difference between the target attention matrix and the full attention matrix meets a preset condition; the full attention matrix is ​​the attention matrix obtained by performing full attention operation on the query feature matrix and the key feature matrix.

[0049] In one possible implementation, the generation unit is specifically configured to perform the following actions: obtaining the product of the first matrix and the query feature matrix to obtain a first intermediate matrix for updating the second matrix; determining the columns and parameter items for updating the second matrix based on the first matrix; and updating the second matrix based on the key feature matrix, the columns and parameter items, and the first intermediate matrix.

[0050] In one possible implementation, the generation unit is specifically configured to perform the following steps: obtain the product of the second matrix and the key feature matrix to obtain a second intermediate matrix when the first matrix is ​​updated; determine the rows and parameter items when the first matrix is ​​updated based on the second matrix; and update the first matrix based on the query feature matrix, the rows and parameter items, and the second intermediate matrix.

[0051] In one possible implementation, each video frame of the video feature sequence includes a first preset number of spatiotemporal blocks, and the first matrix includes sub-matrices corresponding to each spatiotemporal block; wherein, the element value of the kth sub-matrix of the first matrix represents the temporal correlation weight between the kth spatiotemporal block in the i-th video frame and the kth spatiotemporal block in the j-th video frame; wherein k, i, and j are all positive integers.

[0052] In one possible implementation, the video feature sequence includes a second preset number of video frames, and the second matrix includes sub-matrices corresponding to each of the video frames; wherein, the element value of the i-th row and j-th column of the k-th sub-matrix of the second matrix represents the spatial correlation weight between the i-th spatiotemporal block in the k-th video frame and the j-th spatiotemporal block in the k-th video frame of the video feature sequence; wherein k, i, and j are all positive integers.

[0053] In one possible implementation, the generation unit is specifically configured to: obtain a query feature matrix corresponding to a preset frame video frame as a preset frame query feature matrix; perform a full attention operation on the preset frame query feature matrix and the key feature matrix to obtain a preset frame attention matrix; obtain a preset frame attention output based on the preset frame attention matrix and the value feature matrix; and use the preset frame attention output to cover a preset portion of the target attention output to obtain an optimized target attention output; wherein the preset portion is the attention output corresponding to the preset frame video frame in the target attention output.

[0054] According to a third aspect of the present disclosure, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the video generation method as described in the first aspect or any possible implementation thereof.

[0055] According to a fourth aspect of the present disclosure, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the video generation method as described in the first aspect or any possible implementation thereof.

[0056] According to a fifth aspect of the present disclosure, a computer program product is provided, the program product comprising a computer program stored in a readable storage medium, wherein at least one processor of a device reads from the readable storage medium and executes the computer program, causing the device to perform the video generation method described in the first aspect or any possible implementation thereof.

[0057] The technical solution provided by the embodiments of this disclosure brings at least the following beneficial effects: by acquiring a text description and inputting the text description into a pre-trained text-based video model, a target video matching the text description is generated. The text-based video model includes a feature mapping module, an attention module, and a video generation module. The feature mapping module converts the text description into a video feature sequence and maps the video feature sequence into a query feature matrix, a key feature matrix, and a value feature matrix, respectively. The attention module performs a preset attention operation on the query feature matrix and the key feature matrix to obtain a first matrix and a second matrix. The first matrix is ​​used to acquire the correlation information between the same spatiotemporal blocks of different video frames in the video feature sequence. The second matrix is ​​used to acquire the correlation information between different spatiotemporal blocks of the same video frame in the video feature sequence. Then, the attention module obtains a target attention matrix based on the first matrix and the second matrix, and based on the target attention matrix and the value feature matrix... The target attention output is obtained; finally, the video generation module generates the target video based on the target attention output. In this way, the attention module can effectively align the abstract mathematical structure of the target attention matrix with the physical spatiotemporal structure of the video data, decomposing a unified attention computation problem into two sub-problems that specifically handle time and space dependencies. One sub-problem is to determine the correlation information between the same spatiotemporal blocks of different video frames in the video feature sequence; the other sub-problem is to determine the correlation information between different spatiotemporal blocks of the same video frame in the video feature sequence. This allows the text-based video model to involve only relatively small-scale matrix operations in the computation process of generating video content that matches the user's input text description, without having to perform a standard full attention operation for each video frame. This effectively reduces computational and memory complexity, improves computational speed when inferring video content with a large number of frames or high resolution, and effectively enhances video generation efficiency.

[0058] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0059] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0060] Figure 1 This is an application environment diagram illustrating a video generation method according to an exemplary embodiment.

[0061] Figure 2 This is a flowchart illustrating a video generation method according to an exemplary embodiment.

[0062] Figure 3This is a schematic diagram illustrating an attention calculation process according to an exemplary embodiment.

[0063] Figure 4 This is a flowchart illustrating another video generation method according to an exemplary embodiment.

[0064] Figure 5 This is a comparison chart of FLOPs of a video generation method according to an exemplary embodiment.

[0065] Figure 6 This is a comparison chart illustrating the speed improvement of a video generation method according to an exemplary embodiment.

[0066] Figure 7 This is a comparison diagram of the video generation effect of a video generation method according to an exemplary embodiment.

[0067] Figure 8 This is a comparison chart of the runtime of a video generation method according to an exemplary embodiment.

[0068] Figure 9 This is a block diagram illustrating a video generation apparatus according to an exemplary embodiment.

[0069] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0070] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0071] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0072] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0073] The video generation method provided in this disclosure can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with electronic device 104 via a network. A data storage system can store the data that electronic device 104 needs to process. The data storage system can be integrated into electronic device 104 or placed in the cloud or on other network servers.

[0074] In practical applications, terminal 102 can acquire a video generation instruction input by the user and send it to electronic device 104. This video generation instruction can be a video generation prompt, which instructs a pre-trained text-based video model to generate a target video that matches the user's input text description. Specifically, after acquiring the user's input text description, electronic device 104 can input the text description into the pre-trained text-based video model to generate a target video that matches the text description. After obtaining the target video, electronic device 104 can send it to terminal 102, which can then play the target video.

[0075] It should be noted that the pre-trained text-based video model can also be deployed on terminal 102. Terminal 102 can obtain the video generation instructions input by the user, input the text description into the pre-trained text-based video model, and generate a target video that matches the text description. Terminal 102 can then play the target video.

[0076] The pre-trained text-based video model includes a feature mapping module, an attention module, and a video generation module.

[0077] The feature mapping module is used to convert text descriptions into video feature sequences, and then map the video feature sequences into query feature matrices, key feature matrices, and value feature matrices, respectively.

[0078] The attention module is used to perform preset attention operations on the query feature matrix and the key feature matrix to obtain a first matrix and a second matrix; based on the first matrix and the second matrix, a target attention matrix is ​​obtained, and based on the target attention matrix and the value feature matrix, a target attention output is obtained; the first matrix is ​​used to obtain the correlation information between the same spatiotemporal blocks of different video frames in the video feature sequence; the second matrix is ​​used to obtain the correlation information between different spatiotemporal blocks of the same video frame in the video feature sequence.

[0079] The video generation module is used to generate the target video based on the target attention output.

[0080] In practical applications, terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart TVs, smart in-vehicle devices, etc. Portable wearable devices can be head-mounted devices, etc. Electronic device 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0081] Figure 2 This is a flowchart illustrating a video generation method according to an exemplary embodiment, such as... Figure 2 As shown, this video generation method is used for Figure 1 The electronic device 104 includes the following steps.

[0082] In step S202, the text description is obtained.

[0083] The text description can refer to the text used to describe the video content to be generated. For example, the text description could be the user-inputted text "A golden retriever chases butterflies on a sun-drenched grass".

[0084] In practical applications, text descriptions may include, but are not limited to, at least one of video content descriptions, video style descriptions, and video quality descriptions.

[0085] In practice, users can access an AI-based video generation system through a terminal. The terminal can respond to user input of video generation instructions (prompts) to control the system to output videos that meet the user's video generation needs. The terminal can also send video generation instructions to electronic devices, allowing the electronic devices to receive the text descriptions within the instructions.

[0086] In step S204, the text description is input into the pre-trained text-based video model to generate a target video that matches the text description.

[0087] In practice, after obtaining the text description, the electronic device can input the video generation instruction including the text description into the pre-trained text-based video model to trigger the pre-trained text-based video model to generate a target video that matches the text description.

[0088] In practical applications, the pre-trained text-based video model can employ a diffusion model based on the Transformer architecture. In this embodiment, the pre-trained text-based video model may include a feature mapping module, an attention module, and a video generation module.

[0089] The feature mapping module is used to convert text descriptions into video feature sequences and map the video feature sequences into query feature matrices, key feature matrices, and value feature matrices, respectively.

[0090] In its implementation, the feature mapping module includes a pre-trained text encoder and a temporal modeling layer. The electronic device can input text descriptions into the pre-trained text encoder, which converts the text descriptions into text feature vectors. Then, the electronic device can input the text feature vectors into the pre-trained temporal modeling layer, which converts the text feature vectors into a video feature sequence containing video spatiotemporal feature information. In practical applications, the video feature sequence can include T frames, each containing H×W spatiotemporal blocks (tokens). It can be seen that the total number of tokens N in the video feature sequence can be expressed as:

[0091]

[0092] Where H can be the number of spatiotemporal blocks that a single frame of video is divided into in the vertical direction.

[0093] Where W can be the number of spatiotemporal blocks that a single frame of video is divided into in the horizontal width direction.

[0094] In practical applications, the feature mapping module also includes multiple fully connected layers (or convolutional neural network layers). Electronic devices can use multiple fully connected layers to perform dimension mapping on video feature sequences to obtain dimension-matched query feature matrix Q (query matrix), key feature matrix K (key matrix), and value feature matrix V (value matrix) corresponding to the video feature sequence.

[0095] The attention module is used to perform preset attention operations on the query feature matrix and the key feature matrix to obtain a first matrix and a second matrix; based on the first matrix and the second matrix, the target attention matrix is ​​obtained; and based on the target attention matrix and the value feature matrix, the target attention output is obtained.

[0096] The first matrix is ​​used to obtain the correlation information between the same spatiotemporal blocks of different video frames in the video feature sequence.

[0097] The second matrix is ​​used to obtain correlation information between different spatiotemporal blocks of the same video frame in the video feature sequence.

[0098] In specific implementation, after the electronic device obtains the query feature matrix Q, key feature matrix K, and value feature matrix V corresponding to the video feature sequence, the electronic device can input the query feature matrix Q, key feature matrix K, and value feature matrix V into the above attention module to obtain the target attention output.

[0099] In practical applications, electronic devices can use an attention module to perform preset attention operations on the feature matrix and key feature matrix to obtain a first matrix and a second matrix.

[0100] The first matrix L is used to capture temporal correlation (i.e., correlation between frames). In other words, the first matrix L is used to obtain correlation information between the same spatiotemporal blocks of different video frames in the video feature sequence.

[0101] In practical applications, it is assumed that the video feature sequence consists of T frames, each containing If there are b tokens, then the first matrix L can include b tokens of size b. A submatrix. Where m is the time dimension (i.e., the number of frames). b represents the spatial dimension (i.e., the number of tokens per frame). .

[0102] Optionally, each video frame in the video feature sequence includes a first preset number of spatiotemporal blocks (the number of tokens per frame, b). The first matrix L includes sub-matrices corresponding to each spatiotemporal block. The element value in the i-th row and j-th column of the k-th (k=1 to b) sub-matrix of the first matrix L represents the temporal correlation weight between the k-th spatiotemporal block in the i-th video frame and the k-th spatiotemporal block in the j-th video frame. Here, k, i, and j are all positive integers. For example, the element value in the 3-th row and 4-th column of the 5th sub-matrix of the first matrix L represents the association strength between the 5th token in the 3rd video frame and the 5th token in the 4th video frame. The element value can be positively correlated with the association strength.

[0103] Thus, by adopting the first matrix with the above matrix structure, the first matrix has a hardware-friendly data structure, and can efficiently and accurately record the correlation information between the same spatiotemporal blocks of different video frames in the video feature sequence, while facilitating subsequent matrix operations by electronic devices.

[0104] The second matrix R is used to capture spatial correlation (i.e., the association between tokens within a single frame). In other words, the second matrix R is used to obtain correlation information between different spatiotemporal blocks of the same video frame in the video feature sequence.

[0105] In practical applications, if the video feature sequence still includes T frames, and each frame contains H×W tokens, then the second matrix R can include m elements of size ... A submatrix. Where m is the time dimension (i.e., the number of frames). b represents the spatial dimension (i.e., the number of tokens per frame). .

[0106] Optionally, the video feature sequence includes a second preset number of video frames, and the second matrix includes sub-matrices corresponding to each video frame; wherein, the element value of the i-th row and j-th column of the k-th sub-matrix of the second matrix represents the spatial correlation weight between the i-th spatiotemporal block in the k-th video frame and the j-th spatiotemporal block in the k-th video frame; where k, i, and j are all positive integers. For example, the element value of the 5-th row and 8-th column of the 10th sub-matrix of the second matrix R represents the association strength between the 5th token in the 10th video frame and the 8th token in the 10th video frame; wherein, the element value can be positively correlated with the association strength.

[0107] Thus, by adopting the second matrix with the above matrix structure, the second matrix has a hardware-friendly data structure, and can efficiently and accurately record the correlation information between different spatiotemporal blocks of the same video frame in the video feature sequence, while facilitating subsequent matrix operations by electronic devices.

[0108] Then, the electronic device can obtain the target attention matrix based on the first matrix and the second matrix through the attention module, and obtain the target attention output based on the target attention matrix and the value feature matrix.

[0109] In specific implementation, compared to the standard full attention matrix generated based on the query feature matrix Q and the key feature matrix K in traditional technologies, the first matrix L and the second matrix R in this embodiment are two block diagonal matrices with smaller parameters. The electronic device can perform matrix multiplication on the first matrix L, the second matrix R, and the permutation matrix P to obtain the target attention matrix; in other words, the target attention matrix in this embodiment is parameterized as the first matrix L, the second matrix R, and the first permutation matrix. With the second permutation matrix Product. In practical applications, the target attention matrix M can be represented as:

[0110]

[0111] The computational complexity of the target attention matrix M can be expressed as: The computational complexity of the standard full attention matrix can be expressed as: ; Known As can be seen, the technical solution of this embodiment can decompose a huge N×N problem into two smaller subproblems and handle time and space dependent problems separately, which greatly reduces the computational complexity.

[0112] In practical applications, the target attention matrix M can treat multiple frames as a basic temporal unit. For example, set m = T / e, b = e×H×W`, where e is the frame grouping size. This allows for modeling that balances temporal and spatial dependencies at different granularities.

[0113] Then, the electronic device can obtain the target attention output based on the target attention matrix and the value feature matrix. Specifically, the electronic device can perform matrix multiplication on the target attention matrix and the value feature matrix to obtain the initial attention output; then, the electronic device can recompute the initial attention output to obtain the target attention output.

[0114] The video generation module is used to generate the target video based on the target attention output.

[0115] In practice, after the electronic device obtains the target attention output, it can pass the target attention output to the subsequent layers of the text-based video model (i.e., the video generation module). After the subsequent layers go through a multi-step denoising process, the target video is generated.

[0116] In the aforementioned video generation method, a text description is obtained and input into a pre-trained text-based video model to generate a target video matching the text description. This text-based video model includes a feature mapping module, an attention module, and a video generation module. Specifically, the feature mapping module converts the text description into a video feature sequence and maps the video feature sequence into a query feature matrix, a key feature matrix, and a value feature matrix. The attention module performs a preset attention operation on the query feature matrix and the key feature matrix to obtain a first matrix and a second matrix. The first matrix is ​​used to obtain the correlation information between the same spatiotemporal blocks of different video frames in the video feature sequence; the second matrix is ​​used to obtain the correlation information between different spatiotemporal blocks of the same video frame in the video feature sequence. Finally, the attention module obtains the target attention matrix based on the first and second matrices, and obtains the target attention value matrix based on the target attention matrix and the value feature matrix. The attention module outputs the target attention; finally, it generates the target video based on the target attention output. In this way, the attention module can effectively align the abstract mathematical structure of the target attention matrix with the physical spatiotemporal structure of the video data, decomposing a unified attention computation problem into two sub-problems that specifically handle time and space dependencies. One sub-problem is to determine the correlation information between the same spatiotemporal blocks of different video frames in the video feature sequence; the other sub-problem is to determine the correlation information between different spatiotemporal blocks of the same video frame in the video feature sequence. This allows the text-based video model to only involve relatively small-scale matrix operations in the computation process of generating video content that matches the user's input text description, without having to perform a standard full attention operation for each video frame. This effectively reduces computational and memory complexity, improves computational speed when reasoning about video content with a large number of frames, and effectively enhances video generation efficiency.

[0117] In one exemplary embodiment, the number of video frames and the number of spatiotemporal blocks per frame of the video feature sequence are obtained; a permutation matrix is ​​determined based on the number of video frames and the number of spatiotemporal blocks per frame; the permutation matrix includes a first permutation matrix associated with a first matrix and a second permutation matrix associated with a second matrix; and a target attention matrix is ​​generated based on the permutation matrix, the first matrix, and the second matrix.

[0118] The target attention matrix is ​​the product of the first permutation matrix, the first matrix, the second permutation matrix, and the second matrix.

[0119] In specific implementation, during the process of obtaining the target attention matrix based on the first matrix and the second matrix, the electronic device can determine the matrix parameters of the permutation matrix. Specifically, the electronic device can obtain the number of video frames T and the number of spatiotemporal blocks per frame b of the video feature sequence, and determine the permutation matrix based on the number of video frames T and the number of spatiotemporal blocks per frame b. This permutation matrix includes the first permutation matrix associated with the first matrix L. The second permutation matrix associated with the second matrix R .

[0120] Then, the electronic device can manipulate the first matrix L, the second matrix R, and the first permutation matrix. and the second permutation matrix Performing matrix multiplication yields the target attention matrix M, which can be represented as:

[0121]

[0122] The technical solution of this embodiment obtains the number of video frames and the number of spatiotemporal blocks per frame of the video feature sequence, and determines the permutation matrix based on the number of video frames and the number of spatiotemporal blocks per frame; and generates a target attention matrix based on the permutation matrix, the first matrix and the second matrix. The permutation matrix can effectively align the dimensional formats of the first matrix and the second matrix, realize lightweight dimensional rearrangement of the first matrix and the second matrix, and effectively fuse the first matrix and the second matrix. This allows the obtained target attention matrix to accurately capture both inter-frame temporal correlation and intra-frame spatial correlation at the same time, reducing the workload of subsequent electronic devices.

[0123] In one exemplary embodiment, performing a preset attention operation on a query feature matrix and a key feature matrix to obtain a first matrix and a second matrix includes: initializing the first matrix and the second matrix; fixing the first matrix and updating the second matrix based on the first matrix and the key feature matrix; fixing the updated second matrix and updating the first matrix based on the updated second matrix and the query feature matrix; returning to fixing the first matrix and updating the second matrix based on the first matrix and the key feature matrix, until the difference between the target attention matrix and the full attention matrix satisfies a preset condition.

[0124] The full attention matrix is ​​the attention matrix obtained by performing full attention operations on the query feature matrix and the key feature matrix. In other words, the full attention matrix refers to the standard full attention matrix generated based on the query feature matrix Q and the key feature matrix K in traditional techniques. In practical applications, the full attention matrix O can be represented as:

[0125]

[0126] in, Normalized exponential function.

[0127] In specific implementation, such as Figure 3 As shown, in the process of obtaining the first matrix and the second matrix by performing a preset attention operation on the query feature matrix and the key feature matrix, the electronic device employs the following method: Figure 3The attention operation process shown iteratively optimizes the first matrix L and the second matrix R, so that the target attention matrix M, i.e., the Monarch Matrix, obtained by fusing the first matrix L and the second matrix R, can approximate the standard full attention matrix generated based on the query feature matrix Q and the key feature matrix K in traditional techniques. Specifically, the electronic device can initialize the first matrix L and the second matrix R, and solve for the optimal second matrix R while fixing the first matrix L; specifically, the electronic device can update the second matrix R based on the first matrix and the key feature matrix K. Then, the electronic device fixes the second matrix R and solves for the optimal first matrix L; specifically, the electronic device can update the first matrix L based on the updated second matrix R and the query feature matrix Q.

[0128] Then, the electronic device can return to the steps of fixing the first matrix, updating the second matrix based on the first matrix and the key feature matrix, and recording the loop count t, until the difference between the target attention matrix and the full attention matrix meets a preset condition. In practical applications, a good effect can be achieved when the loop count t is greater than a preset threshold. Optionally, the preset threshold can be 2.

[0129] The technical solution of this embodiment involves initializing a first matrix and a second matrix; fixing the first matrix and updating the second matrix based on the first matrix and the key feature matrix; fixing the updated second matrix and updating the first matrix based on the updated second matrix and the query feature matrix; returning to fixing the first matrix and updating the second matrix based on the first matrix and the key feature matrix, until the difference between the target attention matrix and the full attention matrix meets a preset condition. In this way, by iteratively optimizing the first and second matrices based on the alternating minimization algorithm, each step of updating the first and second matrices has a closed-form solution, and the calculation process only involves a small-scale matrix operation, completely avoiding the explicit construction of the standard full attention matrix.

[0130] In one exemplary embodiment, updating the second matrix based on the first matrix and the key feature matrix includes: obtaining the product of the first matrix and the query feature matrix to obtain a first intermediate matrix for updating the second matrix; determining the columns and parameter items for updating the second matrix based on the first matrix; and updating the second matrix based on the key feature matrix, the columns and parameter items, and the first intermediate matrix.

[0131] In specific implementation, such as Figure 3 As shown, during the process of updating the second matrix based on the first matrix and the key feature matrix, the electronic device can obtain the product of the first matrix L and the query feature matrix Q to obtain the first intermediate matrix during the second matrix update. Among them, the first intermediate matrix It can be represented as:

[0132]

[0133] Then, the electronic device can determine the columns and parameter items for updating the second matrix R based on the first matrix L. Among them, columns and parameter items It can be represented as:

[0134]

[0135] Then, the electronic device can be based on the key feature matrix K, columns, and parameter terms. and the first intermediate matrix Update the second matrix R; where the second matrix R can be represented as:

[0136]

[0137] In practical applications, during the update of the second matrix R, it is necessary to calculate a softmax and its entropy. A GPU kernel inspired by the FlashAttention concept can be customized to simultaneously calculate the softmax attention and its Shannon entropy during a single data traversal from HBM to SRAM.

[0138] By maintaining a set of runtime statistics (such as the maximum value of the current block, the sum of the normalized denominators, etc.), the GPU kernel can process data block by block without storing the entire intermediate matrix, which greatly reduces memory usage and data movement overhead, thereby further improving the update speed of the second matrix R and ensuring the performance of this method when processing ultra-long sequences.

[0139] The technical solution of this embodiment obtains the first intermediate matrix for updating the second matrix by multiplying the first matrix and the query feature matrix; and determines the columns and parameter items for updating the second matrix based on the first matrix; and updates the second matrix based on the key feature matrix, columns and parameter items and the first intermediate matrix. This effectively integrates the inter-frame temporal correlations already captured by the first matrix into the update process of the second matrix. By reusing the first matrix to obtain the first intermediate matrix and columns and parameter items required for updating the second matrix, and combining them with the key feature matrix to complete the update of the second matrix, the updated second matrix can accurately capture intra-frame spatial correlations that fit the dynamic logic of the video and match the intent of the text query, while improving the numerical stability of the update process, reducing additional computation, and reducing the operating burden of subsequent electronic devices.

[0140] In one exemplary embodiment, updating the first matrix based on the updated second matrix and the query feature matrix includes: obtaining the product of the second matrix and the key feature matrix to obtain a second intermediate matrix when the first matrix is ​​updated; determining the rows and parameter items when the first matrix is ​​updated based on the second matrix; and updating the first matrix based on the query feature matrix, the rows and parameter items, and the second intermediate matrix.

[0141] For specific implementation details, please refer to [link / reference]. Figure 3 During the process of updating the first matrix based on the updated second matrix and the query feature matrix, the electronic device can obtain the product of the second matrix R and the key feature matrix K to obtain the second intermediate matrix during the update of the first matrix. Among them, the second intermediate matrix It can be represented as:

[0142]

[0143] Then, the electronic device can determine the rows and parameter items for updating the first matrix L based on the second matrix R. Among them, rows and parameter items It can be represented as:

[0144]

[0145] Then, the electronic device can be based on the query feature matrix Q, rows, and parameter items. Second intermediate matrix Update the first matrix L; where the first matrix L can be represented as:

[0146]

[0147] The technical solution of this embodiment obtains the second intermediate matrix when updating the first matrix by multiplying the second matrix and the key feature matrix. Based on the second matrix, the rows and parameter items when updating the first matrix are determined. Then, based on the query feature matrix, the rows and parameter items, and the second intermediate matrix, the first matrix is ​​updated. This can realize the inverse integration of the intra-frame spatial correlation information of the updated second matrix into the update process of the first matrix. By reusing the second matrix, the second intermediate matrix and the rows and parameter items required for updating the first matrix are obtained. Combined with the query feature matrix, the update of the first matrix is ​​completed. This allows the updated first matrix to accurately capture the inter-frame temporal correlation that fits the intra-frame spatial layout and matches the text query intent, while improving the numerical stability of the update process and reducing additional computational overhead, thus reducing the operating pressure of subsequent electronic devices.

[0148] In an exemplary embodiment, the method further includes: obtaining a query feature matrix corresponding to a preset frame video frame as a preset frame query feature matrix; performing a full attention operation on the preset frame query feature matrix and the key feature matrix to obtain a preset frame attention matrix; obtaining a preset frame attention output based on the preset frame attention matrix and the value feature matrix; and using the preset frame attention output to cover a preset portion of the target attention output to obtain an optimized target attention output; wherein the preset portion is the attention output corresponding to the preset frame video frame in the target attention output.

[0149] In specific implementation, for a preset video frame in the video feature sequence, the electronic device can take the query feature matrix corresponding to the preset video frame as the preset frame query feature matrix. Then, the electronic device performs a standard full attention calculation on the preset video frame, that is, performs a full attention operation on the preset frame query feature matrix and key feature matrix to obtain the preset frame attention matrix. Based on the preset frame attention matrix and value feature matrix, the preset frame attention output is obtained. Among them, the preset frame attention output It can be represented as:

[0150]

[0151] For the remaining frames of the video, the target attention output O' is calculated using the above technical solution. The target attention output O' can be expressed as:

[0152]

[0153] Then, the electronic device can use a preset frame attention output. The portion corresponding to the preset video frame in the target attention output O' is covered to form the optimized target attention output.

[0154] The preset frames include at least one of the following: keyframes (such as the first frame of a scene transition), the first x frames (such as the first two frames of a video), the first frame, or the last frame.

[0155] The technical solution of this embodiment integrates the precise correlation information of the full attention operation of the preset frame into the target attention output. By performing full attention operation on the query feature matrix and key feature matrix of the preset frame to obtain the standard preset frame attention output, and covering the part of the preset frame in the target attention output corresponding to the preset frame, the solution effectively solves the attention hole problem that is easily caused by the sparse approximate attention mechanism. This makes the details of the preset frame video clearer and the matching degree with the text description higher. At the same time, since the full attention operation is only performed on the preset frame, the additional computational overhead is minimal and does not affect the overall running efficiency. It can also provide reliable context anchors for subsequent frames, further improving the dynamic coherence and generation quality of the entire video.

[0156] Figure 4 This is a flowchart illustrating another video generation method according to an exemplary embodiment, such as... Figure 3 As shown, this method is used Figure 1 The electronic device in the process includes the following steps.

[0157] In step S402, the text description is obtained, converted into a video feature sequence, and mapped into a query feature matrix, a key feature matrix, and a value feature matrix, respectively.

[0158] In step S404, a preset attention operation is performed on the query feature matrix and the key feature matrix to obtain the first matrix and the second matrix.

[0159] In step S406, the number of video frames and the number of spatiotemporal blocks per frame of the video feature sequence are obtained.

[0160] In step S408, the permutation matrix is ​​determined based on the number of video frames and the number of spatiotemporal blocks per frame.

[0161] In step S410, a target attention matrix is ​​generated based on the permutation matrix, the first matrix, and the second matrix.

[0162] In step S412, the target attention output is obtained based on the target attention matrix and the value feature matrix.

[0163] In step S414, the query feature matrix corresponding to the preset frame video frame is obtained as the preset frame query feature matrix. Full attention operation is performed on the preset frame query feature matrix and the key feature matrix to obtain the preset frame attention matrix. Based on the preset frame attention matrix and the value feature matrix, the preset frame attention output is obtained.

[0164] In step S416, a preset frame attention output is used to cover a preset portion of the target attention output to obtain an optimized target attention output.

[0165] In step S418, the target video is generated based on the target attention output.

[0166] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a video generation method described above, and will not be repeated here.

[0167] The aforementioned video generation method obtains a text description and inputs it into a pre-trained text-based video model to generate a target video matching the text description. This text-based video model includes a feature mapping module, an attention module, and a video generation module. Specifically, the feature mapping module converts the text description into a video feature sequence and maps the video feature sequence into a query feature matrix, a key feature matrix, and a value feature matrix. The attention module performs a preset attention operation on the query feature matrix and the key feature matrix to obtain a first matrix and a second matrix. The first matrix is ​​used to obtain the correlation information between the same spatiotemporal blocks of different video frames in the video feature sequence; the second matrix is ​​used to obtain the correlation information between different spatiotemporal blocks of the same video frame in the video feature sequence. The attention module then obtains a target attention matrix based on the first and second matrices, and obtains the target attention output based on the target attention matrix and the value feature matrix. The target video is generated by the video generation module based on the target attention output. In this way, the attention module effectively aligns the abstract mathematical structure of the target attention matrix with the physical spatiotemporal structure of the video data, decomposing a unified attention computation problem into two sub-problems specifically addressing temporal and spatial dependencies. One sub-problem is determining the correlation information between the same spatiotemporal blocks of different video frames in the acquired video feature sequence; the other sub-problem is determining the correlation information between different spatiotemporal blocks of the same video frame in the video feature sequence. This allows the text-based video model to involve only relatively small-scale matrix operations in the computation process of generating video content that matches the user's input text description, eliminating the need to perform a standard full attention operation for each video frame. This effectively reduces computational and memory complexity, improves computational speed when inferring video content with a large number of frames or high resolution, and significantly enhances video generation efficiency.

[0168] See Figure 5 and Figure 6 Compared with standard full attention, the technical solution disclosed herein can reduce the number of FLOPs (Floating Point Operations per Second) of attention computation by more than 17.5 times, achieving more than 5 times end-to-end operator acceleration in long video generation tasks.

[0169] Meanwhile, by employing spatiotemporal decomposition and pre-defined frame recalculation strategies, efficiency can be significantly improved while generating video quality comparable to, or even superior to, the full attention model in certain metrics, as shown in Table 1. Figure 7 The video generation effect shown.

[0170]

[0171] Table 1

[0172] Furthermore, the technical solution disclosed herein effectively overcomes the attention bottleneck, making it possible to generate longer, higher-resolution videos, and greatly enhancing the scalability of the video generation model. The technical solution disclosed herein also possesses optimized hardware affinity. Based on structured block computation of the target attention matrix and a customized fusion kernel, this method can fully utilize the parallel computing capabilities of modern GPUs, achieving higher practical hardware efficiency than unstructured sparse methods, such as... Figure 8 As shown.

[0173] It should be understood that, although Figure 2 and Figure 4 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 and Figure 4 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0174] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0175] Figure 9 This is a block diagram illustrating a video generation apparatus according to an exemplary embodiment. (Refer to...) Figure 9 The device includes:

[0176] Acquisition unit 910 is configured to acquire text description;

[0177] The generation unit 920 is configured to input the text description into a pre-trained text-based video model to generate a target video that matches the text description; the text-based video model includes a feature mapping module, an attention module, and a video generation module, wherein;

[0178] The feature mapping module is used to convert the text description into a video feature sequence, and map the video feature sequence into a query feature matrix, a key feature matrix and a value feature matrix, respectively;

[0179] The attention module is used to perform a preset attention operation on the query feature matrix and the key feature matrix to obtain a first matrix and a second matrix; the first matrix is ​​used to obtain the correlation information between the same spatiotemporal blocks of different video frames in the video feature sequence; the second matrix is ​​used to obtain the correlation information between different spatiotemporal blocks of the same video frame in the video feature sequence.

[0180] The attention module is further configured to obtain a target attention matrix based on the first matrix and the second matrix, and to obtain a target attention output based on the target attention matrix and the value feature matrix;

[0181] The video generation module is used to generate the target video based on the target attention output.

[0182] In an exemplary embodiment, the generation unit is specifically configured to perform the following operations: acquiring the number of video frames and the number of spatiotemporal blocks per frame of the video feature sequence; determining a permutation matrix based on the number of video frames and the number of spatiotemporal blocks per frame; the permutation matrix including a first permutation matrix associated with the first matrix and a second permutation matrix associated with the second matrix; and generating the target attention matrix based on the permutation matrix, the first matrix, and the second matrix; wherein the target attention matrix is ​​the product of the first permutation matrix, the first matrix, the second permutation matrix, and the second matrix.

[0183] In an exemplary embodiment, the generation unit is specifically configured to perform the following steps: initializing the first matrix and the second matrix; fixing the first matrix and updating the second matrix based on the first matrix and the key feature matrix; fixing the updated second matrix and updating the first matrix based on the updated second matrix and the query feature matrix; returning to the steps of fixing the first matrix and updating the second matrix based on the first matrix and the key feature matrix, until the difference between the target attention matrix and the full attention matrix satisfies a preset condition; the full attention matrix is ​​an attention matrix obtained by performing full attention operations on the query feature matrix and the key feature matrix.

[0184] In an exemplary embodiment, the generation unit is specifically configured to perform the following operations: obtaining the product of the first matrix and the query feature matrix to obtain a first intermediate matrix for updating the second matrix; determining the columns and parameter items for updating the second matrix based on the first matrix; and updating the second matrix based on the key feature matrix, the columns and parameter items, and the first intermediate matrix.

[0185] In an exemplary embodiment, the generation unit is specifically configured to perform the following operations: obtaining the product of the second matrix and the key feature matrix to obtain a second intermediate matrix when the first matrix is ​​updated; determining the rows and parameter items when the first matrix is ​​updated based on the second matrix; and updating the first matrix based on the query feature matrix, the rows and parameter items, and the second intermediate matrix.

[0186] In an exemplary embodiment, each video frame of the video feature sequence includes a first preset number of spatiotemporal blocks, and the first matrix includes sub-matrices corresponding to each spatiotemporal block; wherein, the element value of the kth row and jth column of the kth sub-matrix of the first matrix represents the temporal correlation weight between the kth spatiotemporal block in the i-th video frame and the kth spatiotemporal block in the j-th video frame; wherein k, i, and j are all positive integers.

[0187] In an exemplary embodiment, the video feature sequence includes a second preset number of video frames, and the second matrix includes sub-matrices corresponding to each of the video frames; wherein, the element value of the i-th row and j-th column of the k-th sub-matrix of the second matrix represents the spatial correlation weight between the i-th spatiotemporal block in the k-th video frame and the j-th spatiotemporal block in the k-th video frame of the video feature sequence; wherein k, i, and j are all positive integers.

[0188] In an exemplary embodiment, the generation unit 920 is specifically configured to: obtain a query feature matrix corresponding to a preset frame video frame as a preset frame query feature matrix; perform a full attention operation on the preset frame query feature matrix and the key feature matrix to obtain a preset frame attention matrix; obtain a preset frame attention output based on the preset frame attention matrix and the value feature matrix; and use the preset frame attention output to cover a preset portion of the target attention output to obtain an optimized target attention output; wherein the preset portion is the attention output corresponding to the preset frame video frame in the target attention output.

[0189] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0190] Figure 10 This is a block diagram illustrating an electronic device 1000 for performing a video generation method according to an exemplary embodiment. For example, the electronic device 1000 may be a server. (Refer to...) Figure 10 The electronic device 1000 includes a processing component 1020, which further includes one or more processors, and memory resources represented by memory 1022 for storing instructions, such as application programs, that can be executed by the processing component 1020. The application programs stored in memory 1022 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1020 is configured to execute instructions to perform the methods described above.

[0191] The electronic device 1000 may further include: a power supply component 1024 configured to perform power management of the electronic device 1000, a wired or wireless network interface 1026 configured to connect the electronic device 1000 to a network, and an input / output (I / O) interface 1028. The electronic device 1000 can operate on an operating system stored in memory 1022.

[0192] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1022 including instructions, which can be executed by a processor of an electronic device 1000 to perform the above-described method. The storage medium may be a computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0193] In an exemplary embodiment, a computer program product is also provided, the computer program product including instructions that can be executed by a processor of an electronic device 1000 to perform the above-described method.

[0194] It should be noted that the above-mentioned apparatus, electronic equipment, computer-readable storage medium, computer program product, etc., may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0195] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0196] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video generation method, characterized in that, include: Get the text description; The text description is input into a pre-trained text-based video model to generate a target video that matches the text description; the text-based video model includes a feature mapping module, an attention module, and a video generation module, wherein; The feature mapping module is used to convert the text description into a video feature sequence, and map the video feature sequence into a query feature matrix, a key feature matrix and a value feature matrix, respectively; The attention module is used to perform a preset attention operation on the query feature matrix and the key feature matrix to obtain a first matrix and a second matrix; the first matrix is ​​used to obtain the correlation information between the same spatiotemporal blocks of different video frames in the video feature sequence; the second matrix is ​​used to obtain the correlation information between different spatiotemporal blocks of the same video frame in the video feature sequence. The attention module is further configured to obtain a target attention matrix based on the first matrix and the second matrix, and to obtain a target attention output based on the target attention matrix and the value feature matrix; The video generation module is used to generate the target video based on the target attention output.

2. The video generation method according to claim 1, characterized in that, The step of obtaining the target attention matrix based on the first matrix and the second matrix includes: Obtain the number of video frames and the number of spatiotemporal blocks per frame in the video feature sequence; The permutation matrix is ​​determined based on the number of video frames and the number of spatiotemporal blocks per frame; the permutation matrix includes a first permutation matrix associated with the first matrix and a second permutation matrix associated with the second matrix; The target attention matrix is ​​generated based on the permutation matrix, the first matrix, and the second matrix; The target attention matrix is ​​the product of the first permutation matrix, the first matrix, the second permutation matrix, and the second matrix.

3. The video generation method according to claim 1, characterized in that, The step of performing a preset attention operation on the query feature matrix and the key feature matrix to obtain a first matrix and a second matrix includes: Initialize the first matrix and the second matrix; Fix the first matrix, and update the second matrix based on the first matrix and the key feature matrix; The updated second matrix is ​​fixed, and the first matrix is ​​updated based on the updated second matrix and the query feature matrix; The steps of returning to the fixed first matrix and updating the second matrix based on the first matrix and the key feature matrix continue until the difference between the target attention matrix and the full attention matrix meets a preset condition; the full attention matrix is ​​the attention matrix obtained by performing full attention operation on the query feature matrix and the key feature matrix.

4. The video generation method according to claim 3, characterized in that, The step of updating the second matrix based on the first matrix and the key feature matrix includes: Obtain the product of the first matrix and the query feature matrix to get the first intermediate matrix when updating the second matrix; Based on the first matrix, determine the columns and parameter items for updating the second matrix; The second matrix is ​​updated based on the key feature matrix, the columns and parameter items, and the first intermediate matrix.

5. The video generation method according to claim 3, characterized in that, The step of updating the first matrix based on the updated second matrix and the query feature matrix includes: Obtain the product of the second matrix and the key feature matrix to obtain the second intermediate matrix when updating the first matrix; Based on the second matrix, determine the rows and parameter items for updating the first matrix; The first matrix is ​​updated based on the query feature matrix, the rows and parameter items, and the second intermediate matrix.

6. The video generation method according to any one of claims 1 to 5, characterized in that, Each video frame of the video feature sequence includes a first preset number of spatiotemporal blocks, and the first matrix includes submatrices corresponding to each spatiotemporal block. Wherein, the element value of the kth submatrix of the first matrix represents the temporal correlation weight between the kth spatiotemporal block in the i-th video frame and the kth spatiotemporal block in the j-th video frame of the video feature sequence. Where k, i, and j are all positive integers.

7. The video generation method according to any one of claims 1 to 5, characterized in that, The video feature sequence includes a second preset number of video frames, and the second matrix includes a sub-matrix corresponding to each of the video frames; Wherein, the element value of the kth submatrix of the second matrix represents the spatial correlation weight between the i-th spatiotemporal block in the k-th video frame of the video feature sequence and the j-th spatiotemporal block in the k-th video frame; Where k, i, and j are all positive integers.

8. The video generation method according to any one of claims 1 to 5, characterized in that, The method further includes: Obtain the query feature matrix corresponding to the preset frame video frame, and use it as the preset frame query feature matrix; Perform full attention operation on the preset frame query feature matrix and the key feature matrix to obtain the preset frame attention matrix, and obtain the preset frame attention output based on the preset frame attention matrix and the value feature matrix; The preset frame attention output is used to cover a preset portion of the target attention output to obtain the optimized target attention output; wherein, the preset portion is the attention output corresponding to the preset frame video frame in the target attention output.

9. A video generation apparatus, characterized in that, include: The retrieval unit is configured to retrieve the text description. The generation unit is configured to input the text description into a pre-trained text-based video model and generate a target video that matches the text description; the text-based video model includes a feature mapping module, an attention module, and a video generation module, wherein; The feature mapping module is used to convert the text description into a video feature sequence, and map the video feature sequence into a query feature matrix, a key feature matrix and a value feature matrix, respectively; The attention module is used to perform a preset attention operation on the query feature matrix and the key feature matrix to obtain a first matrix and a second matrix; the first matrix is ​​used to obtain the correlation information between the same spatiotemporal blocks of different video frames in the video feature sequence; the second matrix is ​​used to obtain the correlation information between different spatiotemporal blocks of the same video frame in the video feature sequence. The attention module is further configured to obtain a target attention matrix based on the first matrix and the second matrix, and to obtain a target attention output based on the target attention matrix and the value feature matrix; The video generation module is used to generate the target video based on the target attention output.

10. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the video generation method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the video generation method as described in any one of claims 1 to 8.

12. A computer program product, the computer program product comprising instructions, characterized in that, When the instructions are executed by the processor of the electronic device, the electronic device is able to perform the video generation method as described in any one of claims 1 to 8.

Images