Video generation method, apparatus, device, medium, and product

Abstract

The present disclosure provides a video generation method, device, system, equipment, medium and product, relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, large model and the like. The video generation method comprises: obtaining a difference value between an output feature of a target unit at a current time step and an output feature at a previous time step; in response to the difference value being less than or equal to a preset threshold, taking the output feature at the previous time step as the output feature at the current time step; in response to the difference value being greater than the preset threshold, processing an input feature of a current attention unit based on sparse parameters to obtain an output feature of the current attention unit, and obtaining the output feature at the current time step based on the output feature of the current attention unit; and generating a video based on the output feature at the current time step. The present disclosure can improve the video generation speed and quality.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of computer vision and large models, and specifically to a video generation method, apparatus, system, device, medium, and product. Background Technology

[0002] With technological advancements, video generation models can be used to generate videos from input text. These models are typically based on large, pre-trained models. Summary of the Invention

[0003] This disclosure provides a video generation method, apparatus, system, device, medium, and product.

[0004] According to one aspect of this disclosure, a video generation method is provided, comprising: obtaining a difference value between the output features of a target unit at the current time step and at the previous time step; in response to the difference value being less than or equal to a preset threshold, using the output features of the previous time step as the output features of the current time step; in response to the difference value being greater than the preset threshold, processing the input features of the current attention unit based on sparse parameters to obtain the output features of the current attention unit; obtaining the output features of the current time step based on the output features of the current attention unit; and generating a video based on the output features of the current time step.

[0005] According to another aspect of this disclosure, a video generation apparatus is provided, comprising: an acquisition module, configured to acquire a difference value between the output features of a target unit at the current time step and at the previous time step; a skipping module, configured to, in response to the difference value being less than or equal to a preset threshold, use the output features of the previous time step as the output features of the current time step; a processing module, configured to, in response to the difference value being greater than the preset threshold, process the input features of the current attention unit based on sparse parameters to obtain the output features of the current attention unit, and acquire the output features of the current time step based on the output features of the current attention unit; and a generation module, configured to generate a video based on the output features of the current time step.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein the memory stores instructions executable by said at least one processor, said instructions being executed by said at least one processor to enable said at least one processor to perform the method as described in any of the foregoing aspects.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method according to any of the preceding aspects.

[0008] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to any of the preceding aspects.

[0009] According to embodiments of this disclosure, video generation speed and quality can be improved.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0013] Figure 2 This is a comparative diagram of fixed sparsity and adaptively adjusted sparsity provided according to embodiments of this disclosure;

[0014] Figure 3 This is a schematic diagram according to the second embodiment of the present disclosure;

[0015] Figure 4 This is a schematic diagram of the time-step processing procedure provided according to the embodiments of this disclosure;

[0016] Figure 5 This is a schematic diagram of a Transformer block provided according to an embodiment of this disclosure;

[0017] Figure 6 This is a schematic diagram according to the third embodiment of the present disclosure;

[0018] Figure 7 This is a schematic diagram of an electronic device used to implement the video generation method of the embodiments of this disclosure. Detailed Implementation

[0019] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0020] In the field of generation, diffusion models are typically used as the base model; that is, video generation models can be diffusion models. During the inference phase, diffusion models perform denoising step-by-step, which leads to slow inference speed.

[0021] To improve inference speed, a caching mechanism can be used. That is, for the current time step, when it is determined to skip the current time step, the model output cached in the previous time step is directly used as the model output of the current time step, without performing model calculation.

[0022] In related technologies, it can be determined whether the current time step can be skipped based on the similarity of the input features of adjacent time steps.

[0023] With the development of technology, diffusion models can be based on the Transformer architecture. Under the Transformer architecture, the video generation model includes multiple Transformer blocks. Each Transformer block contains attention layers, feed-forward network (FFN) layers, normalization layers, etc.

[0024] To improve inference speed, a sparse attention mechanism can be employed. This mechanism sparsifies the sequence of tokens used for attention computation based on sparsity rate and a sparse mask, performing attention computation only on key tokens. This reduces computational load and accelerates the inference process. A token is a fundamental computational unit in deep learning, which can be a text unit or an image unit, etc.

[0025] In related technologies, sparse masks can be determined based on randomly sampled Q (query vector) and the full set of K (key vector), and a fixed sparsity rate is usually used.

[0026] The above methods require significant computational and storage overhead, and the quality of the generated video also needs improvement.

[0027] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure, which provides a video generation method. For example... Figure 1 As shown, the method includes:

[0028] 101. Obtain the difference between the output features of the target unit at the current time step and at the previous time step.

[0029] 102. In response to the difference value being less than or equal to a preset threshold, the output feature of the previous time step is used as the output feature of the current time step.

[0030] 103. In response to the difference value being greater than a preset threshold, the input features of the current attention unit are processed based on sparse parameters to obtain the output features of the current attention unit, and the output features of the current time step are obtained based on the output features of the current attention unit.

[0031] 104. Generate a video based on the output features of the current time step.

[0032] The video generation model can be a diffusion model, specifically a diffusion model based on the Transformer architecture.

[0033] In the diffusion model scenario, inference is performed step by step over time. Specifically, in the video generation scenario, video generation is performed step by step over time.

[0034] The total number of time steps can be represented by T (a positive integer), which can be a preset value.

[0035] The current time step refers to the non-first time step currently being executed.

[0036] The target unit refers to a defined unit in a video generation model. For example, if the video generation model is based on the Transformer architecture, which includes multiple Transformer blocks, the first Transformer block can be used as the target unit.

[0037] For the first time step, the usual processing method can be used to obtain the output features of the first time step and cache these output features. The output features of each time step are the output features of the overall model, that is, the final output features of the model after processing by attention layers and FFN layers.

[0038] Taking the target unit as the first Transformer block as an example, for non-first time steps, each non-first time step can be taken as the current time step, and the difference between the output feature of the first Transformer block of the current time step and the output feature of the first Transformer block of the previous time step can be calculated.

[0039] The difference value can be based on the L1 norm, and the calculation formula can be:

[0040]

[0041] in, It is the difference value. It is the output feature of the first Transformer block at the current time step. It is the output feature of the first Transformer block in the previous time step. It is an L1 norm operation.

[0042] When the difference value is less than or equal to a preset threshold, the output features of the previous time step can be directly used as the output features of the current time step without model calculation. This reduces computational load and improves inference speed.

[0043] For example, for the second time step, if the difference between it and the first time step is less than or equal to a preset threshold, the output feature of the cached first time step can be used as the output feature of the second time step.

[0044] When the difference value mentioned above is greater than the preset threshold, the model is calculated for the current time step to obtain the output features of the current time step.

[0045] For example, for the second time step, if the difference between it and the first time step is greater than a preset threshold, the model is executed to calculate the output features of the second time step, and the cached features are updated from the output features of the first time step to the output features of the second time step.

[0046] In the Transformer architecture, model computation includes computation of attention units. To speed up computation, a sparse attention mechanism can be used.

[0047] The current attention unit refers to the attention unit currently being processed within the current time step. Each Transformer block typically includes an attention layer, and each attention layer can include one or more attention heads, each of which can be considered an attention unit.

[0048] Taking the attention unit as the attention head as an example, the sparse parameters of the current attention head can be obtained.

[0049] Sparse parameters include sparsity ratio and sparse mask.

[0050] Sparsity ratio indicates the proportion of non-critical token pairs. The larger the value, the fewer token pairs need to be computed. For example, if the total number of input tokens in the current attention head is N (a positive integer), attention calculation typically requires N*N attention channels (* represents multiplication). Assuming a sparsity ratio of 80%, only 0.2*N*N attention channels need to be computed, thus reducing computational load and improving inference speed.

[0051] A sparse mask is used to indicate the positional relationship between token pairs participating in attention calculation. Based on the example above, a sparse mask is typically an N*N matrix, where each element is either 0 or 1. 1 indicates that the token at the corresponding position is subject to attention calculation, while 0 indicates that the token at the corresponding position is not subject to attention calculation.

[0052] For the same time step, different attention units within that time step can use the same sparsity rate, and the sparse masks of different attention units can be the same or different.

[0053] After obtaining the sparse parameters of the current attention unit, the input features of the current attention unit are sparsified using these sparse parameters to obtain the output features of the current attention unit.

[0054] Then, the output features of the current time step can be obtained based on the output features of the current attention unit. For example, in addition to the attention layer, the video generation model may also include FFN, etc. Therefore, the output features of the attention layer can be processed by FFN, etc., to obtain the output of the overall model, which can be used as the output features of the current time step.

[0055] After obtaining the output features of the current time step, a video can be generated based on the output features of the current time step. For example, by executing a preset number of time steps, the output features of the last time step can be obtained, and the output features can be decoded and converted into a video.

[0056] In this embodiment, when the difference value is less than or equal to a preset threshold, the output feature of the previous time step is used as the output feature of the current time step. When the difference value is greater than the preset threshold, sparse attention processing is performed. This can combine caching and sparse attention mechanisms to improve video generation speed. In addition, the difference value is determined based on the output feature of the target unit, which can improve the accuracy of the difference value and thus improve the quality of video generation, such as improving the content consistency of different frame images.

[0057] In some embodiments, the sparse parameter includes: a sparse mask;

[0058] The method further includes:

[0059] Based on the input features, determine the query vector and the key vector;

[0060] Based on the query vector and the key vector, determine the attention matrix;

[0061] The sparse mask is determined based on the attention matrix.

[0062] Sparse masks can be categorized into time-dimensional masks or spatial-dimensional masks.

[0063] In related technologies, a preset number of tokens corresponding to Q are usually randomly sampled. The attention output (represented by Output0) is calculated using the randomly selected Q and the complete K. Then, the temporal dimension mask and the spatial dimension mask are used in the attention calculation process to obtain the corresponding outputs (represented by Output1 and Output2). The L2 loss of Output0, Output1 and Output2 is calculated. The temporal dimension mask or the spatial dimension mask is selected based on the minimum L2 loss.

[0064] However, the above methods require a large amount of computation and storage space, resulting in a waste of resources.

[0065] In this embodiment, Q and K are determined based on the input features, and then an attention matrix is ​​determined based on Q and K. A sparse mask is then determined based on the attention matrix. This method can reduce the amount of computation and data, thereby reducing computational and storage overhead.

[0066] In some embodiments, the sparse mask includes: a temporal dimension mask, or a spatial dimension mask; determining the sparse mask based on the attention matrix includes:

[0067] In response to the attention matrix being a diagonal matrix, the current mask is determined to be a time-dimensional mask; otherwise, the current mask is determined to be a spatial-dimensional mask.

[0068] The attention matrix can be calculated based on the query vector Q and the key vector K, such as attention matrix = Q*K. T , * indicates multiplication, and the superscript T indicates transpose.

[0069] After calculating the attention matrix, if it is a diagonal matrix, the sparse mask is determined to be a temporal dimension mask; otherwise, if it is a non-diagonal matrix, the sparse mask is determined to be a spatial dimension mask. A diagonal matrix is ​​defined as a matrix where the diagonal elements are 1s and the remaining elements are 0s. In practice, the proportion of diagonal elements with 1s in the attention matrix can be obtained; if this proportion is greater than a threshold, the attention matrix is ​​also considered a diagonal matrix.

[0070] In this embodiment, the sparse mask is determined based on whether the attention matrix is ​​a diagonal matrix, which can efficiently and accurately determine the sparse mask and improve processing efficiency.

[0071] In some embodiments, the input features are multiple frames;

[0072] The process of determining the query vector and key vector based on the input features includes:

[0073] The query vector is determined based on the input features of the first frame;

[0074] The key vector is determined based on the input features of the last frame.

[0075] In the video generation scenario, the input features of the attention unit include the input features of multiple frames in time. The first frame is earlier than the last frame in time. When determining Q and K, Q is calculated based on the input features of the first frame, and K is calculated based on the input features of the last frame. Then, the sparse mask is determined based on Q and K.

[0076] Furthermore, Q and K mentioned above can also be obtained based on tokens at fixed spatial locations. For example, the input features of the attention unit are M*C dimensional, where M and C are both positive integers, C is the total number of frames, and M is the number of tokens per frame. After mapping the input features, initial Q and initial K can be obtained, each corresponding to M tokens. Then, features corresponding to a preset number and spatial location of tokens can be selected from the M tokens as target Q and target K, respectively. For example, the features corresponding to the first 64 tokens of the first frame can be used as target Q, and the features corresponding to the first 64 tokens of the last frame can be used as target K. The attention matrix is ​​calculated using target Q and target K. Based on whether the attention matrix is ​​a diagonal matrix, it is determined whether the sparse mask is a temporal dimension mask or a spatial dimension mask.

[0077] In this embodiment, Q is calculated based on the input features of the first frame, and K is calculated based on the input features of the last frame. By taking into account temporal information, accurate Q and K can be obtained, thereby improving the accuracy of the sparse mask.

[0078] The sparsity parameters mentioned above also include the sparsity rate. Initially, the initial sparsity rate can be set based on experience, and then, at the target time step, the sparsity rate can be dynamically adjusted as needed.

[0079] In some embodiments, the method further includes:

[0080] In response to the current time step being the target time step, the proportion of the target mask within the current time step is determined;

[0081] The sparsity rate is adjusted based on the stated proportion.

[0082] In the intermediate time step between the first and last time step, one or more can be selected as target time steps according to preset rules. At the target time step, the proportion of the target mask can be counted, and the sparsity rate can be adjusted according to the proportion. Subsequent time steps will use the adjusted sparsity rate.

[0083] For example, for the current time step, the existing sparsity rate can be obtained. Sparse attention processing can then be performed based on this sparsity rate. After completing the sparse attention processing for the current time step, the proportions of the temporal and spatial dimension masks for that time step can be calculated. For instance, if the current time step includes n attention units, and m of these attention units use temporal dimension masks, then the proportion of the temporal dimension mask = m / n, and the proportion of the spatial dimension mask = (nm) / m. This proportion can then be used to adjust the sparsity rate.

[0084] In this embodiment, by adjusting the sparsity rate based on the proportion, the sparsity rate can be adaptively adjusted, thereby improving the quality of the generated video, such as enhancing the consistency of content among different images in the video.

[0085] The target mask is preset, such as using the time dimension mask as the target mask, and then adjusting the sparsity based on the proportion of the time dimension mask.

[0086] In some embodiments, adjusting the sparsity rate based on the proportion includes:

[0087] In response to the target mask being a time-dimensional mask, the sparsity is adjusted based on a preset inverse mapping relationship.

[0088] Specifically, the proportion of the time dimension mask is inversely related to the sparsity; that is, the higher the proportion of the time dimension mask, the lower the sparsity. The adjusted sparsity can be obtained based on the time dimension mask using a preset inverse relationship function.

[0089] After obtaining the adjusted sparsity, subsequent time steps use this adjusted sparsity for sparse attention processing until the sparsity is readjusted again.

[0090] Since the caching mechanism adaptively determines the time steps that can be skipped, combining this caching mechanism with the sparse attention mechanism may cause transient phenomena in the middle of the video, resulting in inconsistencies between the first few frames and the last few frames in terms of the subject and background, thus affecting the quality of the generated video.

[0091] In this embodiment, when the proportion of the temporal dimension mask is high, the sparsity is reduced to retain more temporal correlation information; when this proportion is low, the sparsity is increased to reduce redundant computation and generation failures. By adaptively adjusting the sparsity rate, the aforementioned transient phenomenon can be reduced, content consistency can be improved, and thus the video generation quality can be effectively improved.

[0092] Figure 2 This is a comparative diagram of a fixed sparsity rate and an adaptively adjusted sparsity rate provided according to embodiments of this disclosure.

[0093] like Figure 2 As shown, the top image is generated using a fixed sparsity rate, while the bottom image is generated using an adaptive sparsity rate. The comparison reveals that the video generated using a fixed sparsity rate may exhibit inconsistencies, such as the subject being highlighted appearing differently in different frames. In contrast, the video generated using an adaptive sparsity rate maintains consistency between the subject and background across different frames, effectively improving the overall video quality.

[0094] In conjunction with the above, this disclosure may also provide the following embodiments.

[0095] Figure 3 This is a schematic diagram based on a second embodiment of the present disclosure, which provides a video generation method. For example... Figure 3 As shown, the method includes:

[0096] 301. At the first time step, perform model calculations on the input information to obtain the output features of the first time step, and cache the output features.

[0097] In the case of a diffusion model for video generation, feature processing is performed step by step over time.

[0098] Figure 4 This is a schematic diagram of the time-step processing procedure provided according to the embodiments of this disclosure.

[0099] like Figure 4 As shown, in a text-based video scenario, the input information includes text features, time-step features, and input features. Text features are obtained by characterizing the user-input prompt, while time-step features are obtained by characterizing the timestamp of the current time step. For the first time step (t=1), the input features are the initial noise features. In subsequent time steps (such as t=2 and t=3), the output features of the previous time step can be used as the input features for the current time step.

[0100] For the first time step, the output features of the first time step need to be cached.

[0101] 302. Obtain the difference between the output features of the target unit at the current time step and at the previous time step.

[0102] In this process, each non-first time step is taken as the current time step, and the difference between the output features of the target unit at the current time step and at the previous time step is calculated.

[0103] For example, if the target unit is the first Transformer block, then at the second time step (t=2), the difference between the output feature of the first Transformer block at the second time step and the output feature of the first Transformer block at the first (first) time step is calculated. Here, the first Transformer block refers to the first Transformer block into which the input feature enters. Starting from the first Transformer block, the input feature is processed block by block to obtain the output feature at the current time step.

[0104] 303. In response to the difference value being less than or equal to a preset threshold, the output feature of the previous time step is used as the output feature of the current time step.

[0105] For example, such as Figure 4 As shown, assuming that the difference between the second time step (t=2) and the first time step (t=1) is less than or equal to a preset threshold, the output feature of the first time step is used as the output feature of the second time step. That is, the non-first Transformer block of the second time step is no longer calculated.

[0106] 304. In response to the difference value being greater than a preset threshold, a query vector is determined based on the input features of the first frame, a key vector is determined based on the input features of the last frame, and a sparse mask is determined based on the query vector and the key vector.

[0107] For example, such as Figure 4 As shown, assuming that the difference between the third time step (t=3) and the second time step (t=2) is greater than the preset threshold, the third time step needs to perform model calculation, that is, the input features of the time step are calculated from the first Transformer block to the last Transformer block to obtain the output features of the time step.

[0108] For non-skipped time steps, update the cached value with the output features of that time step. For example, in the cache, update the output features of the first time step with the output features of the third time step. Then, in subsequent time steps, the output features of the third time step can be directly obtained from the cache and used as the output features for subsequent skippable time steps.

[0109] For each time step where model computation is required, computation is performed sequentially based on each Transformer block within that time step.

[0110] Figure 5 This is a schematic diagram of a Transformer block provided according to an embodiment of this disclosure.

[0111] like Figure 5As shown, each Transformer block may include an attention layer, FFN, etc. For this purpose, a sparse attention mechanism can be used for the attention layer.

[0112] For each Transformer block, in a video generation scenario, its input features include multiple frames in time sequence. A query vector Q is calculated based on the input features of the earliest frame, and a key vector K is calculated based on the input features of the latest frame. Then, a sparse mask is determined based on Q and K. For example, a preset query mapping matrix is ​​used to map the input features of the first frame to obtain an initial Q, which includes features corresponding to multiple tokens. The features corresponding to the token at a preset spatial location are then selected as the target Q. Similarly, a preset key mapping matrix is ​​used to map the input features of the last frame to obtain an initial K, which also includes features corresponding to multiple tokens. The features corresponding to the token at the aforementioned spatial locations are then selected as the target K. An attention matrix is ​​calculated between the target Q and target K. If this attention matrix is ​​diagonal, a temporal sparse mask is used; otherwise, a spatial sparse mask is used.

[0113] Alternatively, you can first obtain the sparsity rate, and then select a time-dimension sparse mask or a spatial-dimension sparse mask that matches that sparsity rate. The initial value of the sparsity rate is preset, and can be adjusted later according to the actual situation.

[0114] 305. Based on the sparse mask, process the input features of the current attention unit to obtain the output features of the current attention unit, and based on the output features of the current attention unit, obtain the output features of the current time step.

[0115] For example, taking the current attention unit as an attention layer as an example, such as Figure 5 As shown, after calculating the attention layer based on the sparse mask, the output feature of the attention layer is obtained. Then, after calculating the output feature of the FFN layer, the output feature of the current Transformer block is obtained. After calculating each Transformer block in sequence, the output feature of the last Transformer block is obtained, which is used as the output feature of the current time step.

[0116] 306. In response to the current time step being the target time step, adjust the sparsity rate based on the proportion of the target mask within the current time step.

[0117] One or more target time steps can be selected from those that are neither the first nor the last time step. The sparsity can also be adjusted within the target time step.

[0118] Specifically, when the target time step is a non-skipped time step, the proportion of its target mask (such as a time-dimensional sparse mask) can be counted, and the sparsity rate can be adjusted according to this proportion. For example, the higher the proportion of the time-dimensional sparse mask, the lower the sparsity rate.

[0119] 307. Based on the output features of the current time step and the adjusted sparsity, process subsequent time steps until the output features of the last time step are obtained. Decode the input features of the last time step to obtain the video.

[0120] For example, refer to Figure 4 After executing each time step in sequence, the output features of the last time step can be obtained. After decoding the output features, the features can be converted into video, thus enabling video generation based on text information (prompt).

[0121] In this embodiment, a caching mechanism and a sparse attention mechanism are combined to speed up video generation and improve video quality by adaptively adjusting the sparsity rate.

[0122] Figure 6 The schematic diagram is based on the third embodiment of the present disclosure. This embodiment provides a video generation device 600, which includes: an acquisition module 601, a skipping module 602, a processing module 603, and a generation module 604.

[0123] The acquisition module 601 is used to acquire the difference value between the output features of the target unit at the current time step and at the previous time step; the skip module 602 is used to, in response to the difference value being less than or equal to a preset threshold, use the output features of the previous time step as the output features of the current time step; the processing module 603 is used to, in response to the difference value being greater than the preset threshold, process the input features of the current attention unit based on sparse parameters to obtain the output features of the current attention unit, and acquire the output features of the current time step based on the output features of the current attention unit; the generation module 604 is used to generate a video based on the output features of the current time step.

[0124] In this embodiment, when the difference value is less than or equal to a preset threshold, the output feature of the previous time step is used as the output feature of the current time step. When the difference value is greater than the preset threshold, sparse attention processing is performed. This can combine caching and sparse attention mechanisms to improve video generation speed. In addition, the difference value is determined based on the output feature of the target unit, which can improve the accuracy of the difference value and thus improve the quality of video generation.

[0125] In some embodiments, the sparse parameter includes: a sparse mask;

[0126] The device further includes:

[0127] The determination module is used to determine a query vector and a key vector based on the input features; determine an attention matrix based on the query vector and the key vector; and determine the sparse mask based on the attention matrix.

[0128] In this embodiment, Q and K are determined based on the input features, and then an attention matrix is ​​determined based on Q and K. A sparse mask is then determined based on the attention matrix. This method can reduce the amount of computation and data, thereby reducing computational and storage overhead.

[0129] In some embodiments, the sparse mask includes: a temporal dimension mask, or a spatial dimension mask; the determining module is further configured to:

[0130] In response to the attention matrix being a diagonal matrix, the sparse mask is determined to be a time-dimensional mask; otherwise, the sparse mask is determined to be a spatial-dimensional mask.

[0131] In this embodiment, the sparse mask is determined based on whether the attention matrix is ​​a diagonal matrix, which can efficiently and accurately determine the sparse mask and improve processing efficiency.

[0132] In some embodiments, the input features are multiple frames; the determining module is further configured to:

[0133] The query vector is determined based on the input features of the first frame;

[0134] The key vector is determined based on the input features of the last frame.

[0135] In this embodiment, Q is calculated based on the input features of the first frame, and K is calculated based on the input features of the last frame. By taking into account temporal information, accurate Q and K can be obtained, thereby improving the accuracy of the sparse mask.

[0136] In some embodiments, the sparsity parameter includes: sparsity ratio; the device further includes:

[0137] An adjustment module is used to determine the proportion of the target mask within the current time step in response to the current time step being the target time step; and adjust the sparsity rate based on the proportion.

[0138] In this embodiment, by adjusting the sparsity based on the proportion, the sparsity can be adaptively adjusted, thereby improving the quality of the generated video.

[0139] In some embodiments, the adjustment module is further configured to:

[0140] In response to the target mask being a time-dimensional mask, the sparsity is adjusted based on a preset inverse mapping relationship.

[0141] In this embodiment, when the proportion of the temporal dimension mask is high, the sparsity is reduced to retain more temporal correlation information; when this proportion is low, the sparsity is increased to reduce redundant computations and generation failures. By adaptively adjusting the sparsity rate, the video generation quality can be effectively improved.

[0142] It is understood that the same or similar content in different embodiments of this disclosure can be referred to each other.

[0143] It is understood that the terms "first" and "second" in the embodiments of this disclosure are only used for distinction and do not indicate the degree of importance or the order of events.

[0144] It is understandable that, unless otherwise specified, the order of steps in the process indicates that the temporal relationship between these steps is not limited.

[0145] The collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0146] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0147] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device 700 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0148] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 707 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0149] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0150] The computing unit 701 can be a variety of general-purpose and / or proprietary processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various proprietary artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as video generation methods. For example, in some embodiments, the video generation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the video generation method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the video generation method by any other suitable means (e.g., by means of firmware).

[0151] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), proprietary integrated circuits (ASICs), proprietary standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a proprietary or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programmable task processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0153] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0154] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0155] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0156] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0157] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0158] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video generation method, comprising: Obtain the difference value between the output features of the target unit at the current time step and at the previous time step; In response to the difference value being less than or equal to a preset threshold, the output feature of the previous time step is used as the output feature of the current time step; In response to the difference value being greater than a preset threshold, the input features of the current attention unit are processed based on sparse parameters to obtain the output features of the current attention unit, and the output features of the current time step are obtained based on the output features of the current attention unit. A video is generated based on the output features of the current time step.

2. The method according to claim 1, wherein, The sparse parameters include: sparse mask; The method further includes: Based on the input features, determine the query vector and the key vector; Based on the query vector and the key vector, determine the attention matrix; The sparse mask is determined based on the attention matrix.

3. The method according to claim 2, wherein, The sparse mask includes: a time-dimensional mask, or a spatial-dimensional mask; Determining the sparse mask based on the attention matrix includes: In response to the attention matrix being a diagonal matrix, the sparse mask is determined to be a time-dimensional mask; otherwise, the sparse mask is determined to be a spatial-dimensional mask.

4. The method according to claim 2, wherein, The input features are multiple frames; The process of determining the query vector and key vector based on the input features includes: The query vector is determined based on the input features of the first frame; The key vector is determined based on the input features of the last frame.

5. The method according to claim 1, wherein, The sparsity parameter includes: sparsity ratio; The method further includes: In response to the current time step being the target time step, the proportion of the target mask within the current time step is determined; The sparsity rate is adjusted based on the stated proportion.

6. The method according to claim 5, wherein, The adjustment of the sparsity rate based on the ratio includes: In response to the target mask being a time-dimensional mask, the sparsity is adjusted based on a preset inverse mapping relationship.

7. A video generation apparatus, comprising: The acquisition module is used to acquire the difference value between the output features of the target unit at the current time step and the previous time step; The skip module is used to, in response to the difference value being less than or equal to a preset threshold, use the output feature of the previous time step as the output feature of the current time step; The processing module is used to process the input features of the current attention unit based on sparse parameters in response to the difference value being greater than a preset threshold, to obtain the output features of the current attention unit, and to obtain the output features of the current time step based on the output features of the current attention unit. The generation module is used to generate a video based on the output features of the current time step.

8. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images