Video generation method, apparatus, device, and storage medium

Abstract

The present disclosure provides a video generation method, device, equipment and storage medium. By adding a space-time downsampling module in the video generation model, the fusion features of each video frame are downsampled from the spatial dimension before the inverse diffusion of the video generation model, and the fusion feature sequence is downsampled from the time sequence dimension, the feature data is compressed, the data amount of the video frame hidden feature in the inverse diffusion process is reduced, the generated video frame number can be improved in the case of limited computing resources, thereby increasing the generated video, and in the model training stage, the features of the sample video are compressed in space and time, the sample data amount is reduced, and long videos can be directly used for model training to improve the video generation quality.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to artificial intelligence technology, and in particular to a video generation method, device, equipment and storage medium. BACKGROUND

[0002] With the continuous development of artificial intelligence (AI) technology, AI video generation has made significant progress in recent years. Long video generation, as an important task in the field of computer vision, has attracted the attention of many researchers. Long video generation is an important research direction of video generation technology, and long video can better simulate complex scenes and events in the real world, and has a wide range of applications.

[0003] Due to the limitation of computing resources, it is difficult for existing technologies to directly use long videos for model training, so researchers have explored optimization techniques and novel algorithms to improve the long video generation effect under limited computing resources. Current long video generation techniques can be roughly divided into two categories: divide-and-conquer and autoregressive. The divide-and-conquer method first generates key frames, then generates intermediate frames, iteratively refines, and finally obtains a long video; the autoregressive method uses the last few frames of a video as a generation condition to generate the next video, and iteratively generates a long video.

[0004] However, the long video generated by the divide-and-conquer method can increase the number of frames, but cannot extend the video content and can only be limited to the content corresponding to the key frames. Although the autoregressive method can theoretically generate an infinite long video, the error accumulation problem existing in the autoregressive method will cause the quality of the generated video to degrade. SUMMARY

[0005] The embodiments of the present disclosure provide a video generation method, device, equipment and storage medium, which can increase the length of the generated video and improve the video quality.

[0006] In one aspect of the embodiments of the present disclosure, a video generation method is provided, comprising:

[0007] In response to a video generation instruction, a first frame image and a description text specified by the video generation instruction are obtained, and the description text is used to indicate video content;

[0008] The first frame image and the description text are respectively subjected to feature extraction to obtain first frame features and text features;

[0009] perform feature fusion on the first frame feature, the text feature, and random noise, input a fusion feature sequence obtained by the feature fusion into a video generation model, and generate a target hidden feature sequence corresponding to a video frame sequence in a hidden space through the video generation model; wherein the video generation model comprises a space-time downsampling module and an inverse diffusion module, the space-time downsampling module is configured to downsample fusion features of each video frame in a spatial dimension, and downsample the fusion feature sequence in a time sequence dimension, and the inverse diffusion module is configured to perform inverse diffusion processing based on an initial feature sequence after downsampling;

[0010] input the target hidden feature sequence into a video decoder, and generate a target video corresponding to the target hidden feature sequence through the video decoder.

[0011] Optionally, the generating, by the video generation model, of the target hidden feature sequence corresponding to the video frame sequence in the hidden space comprises:

[0012] downsampling, by the space-time downsampling module, the fusion feature sequence to obtain a downsampled fusion feature sequence;

[0013] performing, by the inverse diffusion module, inverse diffusion processing on the downsampled fusion feature sequence to obtain a downsampled hidden feature sequence;

[0014] performing, by a space-time upsampling module of the video generation model, space-time upsampling on the downsampled hidden feature sequence to obtain the target hidden feature sequence.

[0015] Optionally, the downsampling, by the space-time downsampling module, of the fusion feature sequence to obtain a downsampled fusion feature sequence comprises:

[0016] downsampling, by the space-time downsampling module, the fusion feature to obtain a local downsampled feature;

[0017] performing, by a first attention network, self-attention calculation on the local downsampled feature, and performing cross-attention calculation on the local downsampled feature and the text feature to obtain the downsampled fusion feature sequence;

[0018] the performing, by the space-time upsampling module of the video generation model, of the space-time upsampling on the downsampled hidden feature sequence to obtain the target hidden feature sequence comprises:

[0019] performing, by the space-time upsampling module, space-time upsampling on the downsampled hidden feature sequence to obtain a local hidden feature sequence;

[0020] The second attention network is used for self-attention calculation on the local hidden feature, and cross-attention calculation is performed on the local hidden feature and the text feature to obtain the target hidden feature sequence.

[0021] Optionally, the spatial-temporal down-sampling module includes a spatial down-sampling unit and a time down-sampling unit, and the spatial-temporal up-sampling module includes a spatial up-sampling unit and a time up-sampling unit.

[0022] The spatial down-sampling unit is configured to perform spatial down-sampling on the fusion feature in the fusion feature sequence through a two-dimensional convolution layer with a preset convolution kernel size and a preset step length.

[0023] The time down-sampling unit is configured to perform time down-sampling on the fusion feature sequence through a three-dimensional average pooling layer with a preset convolution kernel size and a preset step length.

[0024] The spatial up-sampling unit is configured to perform spatial up-sampling on the down-sampled hidden feature sequence through spatial nearest-neighbor interpolation.

[0025] The time up-sampling unit is configured to perform time up-sampling on the down-sampled hidden feature sequence through time nearest-neighbor interpolation.

[0026] Optionally, the method further includes:

[0027] obtaining structure information specified by the video generation instruction, the structure information including key point coordinates of the target object in each video frame;

[0028] extracting features of the structure information through a structure encoder to obtain structure features;

[0029] The feature fusion of the first frame feature, the text feature and the random noise is performed, and the obtained fusion feature sequence is input into a video generation model, and the target hidden feature sequence corresponding to the video frame sequence in the hidden space is generated by the video generation model.

[0030] The feature fusion of the first frame feature, the text feature, the structure feature and the random noise is performed, and the obtained fusion feature sequence is input into the video generation model to generate the target hidden feature sequence; and the video decoder is configured to generate the target video in which the target object moves in a movement manner indicated by the structure information based on the target hidden feature sequence and the video generation model.

[0031] Optionally, before the first frame image and the description text specified by the video generation instruction are obtained in response to the video generation instruction, the method includes:

[0032] The model training framework includes a video encoder, the video generation model, the video decoder, a first frame encoder, and a text encoder, the video encoder and the video generation model each include the spatio-temporal up-sampling module, and the video generation model and the video decoder each include a spatio-temporal down-sampling module;

[0033] The parameters of the video generation model are fixed, and the video encoder and the video decoder are trained by using a sample video;

[0034] In response to the training of the video encoder and the video decoder being completed, the parameters of the video encoder and the video decoder are fixed, and the video generation model is trained by using the sample video.

[0035] Optionally, the training of the video encoder and the video decoder by using the sample video comprises:

[0036] The sample video is processed by using the video encoder to obtain a mean value and a standard deviation of a sample hidden feature sequence;

[0037] The sample hidden feature sequence is obtained by performing parameter renormalization based on the mean value and the standard deviation;

[0038] The sample hidden feature sequence is processed by using the video decoder to obtain a reconstructed video;

[0039] The parameters of the video encoder and the video decoder are updated based on the sample video and the reconstructed video.

[0040] Optionally, the updating of the parameters of the video encoder and the video decoder based on the sample video and the reconstructed video comprises:

[0041] A first loss function value is obtained by calculating a function value of the first loss function based on the sample video and the reconstructed video by using a first loss function;

[0042] The parameters of the video encoder and the video decoder are updated based on the first loss function value;

[0043] The first loss function is:

[0044] L = L rec + 0.000001·L kl

[0045]

[0046] L kl = -0.5(logσ2 +1-μ 2 -σ 2 )

[0047] Where σ is the standard deviation of the latent feature sequence of the sample, and μ is the mean of the latent feature sequence of the sample. y is the output of the video decoder, and y is the input of the video encoder.

[0048] Optionally, training the video generation model using the sample video includes:

[0049] The trained video encoder is used to perform video encoding and spatiotemporal downsampling on the sample video to obtain the sample latent feature sequence;

[0050] The sample latent feature sequence is noise-added using sample noise, and then fused with the sample first frame feature and sample text feature to obtain the sample fused feature sequence. The sample first frame feature is obtained by feature extraction and spatiotemporal downsampling of the first frame image of the sample video using a first frame encoder, and the sample text feature is obtained by feature extraction of the sample description text of the sample video using a text encoder.

[0051] The video generation model is used to perform spatiotemporal downsampling and inverse diffusion processing on the sample fusion feature sequence to obtain prediction noise;

[0052] The parameters of the video generation model are updated based on the sample noise and the predicted noise.

[0053] Optionally, updating the parameters of the video generation model based on the sample noise and the predicted noise includes:

[0054] Using the second loss function, the function value of the second loss function is calculated based on the sample noise and the prediction noise to obtain the value of the second loss function;

[0055] The parameters of the video generation model are updated based on the value of the second loss function;

[0056] The second loss function is:

[0057]

[0058] Where, ε t For the sample noise added in the t-th iteration, x t Let be the latent feature sequence of the sample after the t-th noise addition, c be the control condition including the first frame features of the sample and the text features of the sample, and ε be the latent feature sequence of the sample after the t-th noise addition. θ (x tt, c) is predicted noise obtained by the video generation model predicting the sample noise added for the t th time.

[0059] In another aspect of the embodiments of the present disclosure, a video generation device is provided, comprising:

[0060] An information acquisition module is configured to acquire, in response to a video generation instruction, a first frame image and a description text specified by the video generation instruction, the description text being used to indicate video content.

[0061] A feature extraction module is configured to perform feature extraction on the first frame image and the description text respectively to obtain a first frame feature and a text feature.

[0062] A feature generation module is configured to perform feature fusion on the first frame feature, the text feature, and random noise, input a fusion feature sequence obtained into a video generation model, and generate, by the video generation model, a target hidden feature sequence corresponding to a video frame sequence in a hidden space; wherein the video generation model comprises a space-time down-sampling module and an inverse diffusion module, the space-time down-sampling module is configured to down-sample the fusion feature of each video frame in a spatial dimension and down-sample the fusion feature sequence in a time sequence dimension, and the inverse diffusion module is configured to perform inverse diffusion processing based on an initial feature sequence after down-sampling.

[0063] A video generation module is configured to input the target hidden feature sequence into a video decoder, and generate a target video corresponding to the target hidden feature sequence by the video decoder.

[0064] In another aspect of the embodiments of the present disclosure, an electronic device is provided, comprising:

[0065] A memory is configured to store a computer program.

[0066] A processor is configured to execute the computer program stored in the memory, and when the computer program is executed, the method in the above aspect is implemented.

[0067] In another aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the method in the above aspect is implemented.

[0068] In another aspect of the embodiments of the present disclosure, a computer program is provided, comprising computer program instructions, and when the computer program instructions are executed by a processor, the method in the above aspect is implemented.

[0069] Based on the embodiments of the present disclosure, by adding a space-time down-sampling module in the video generation model, the fusion features of each video frame are down-sampled from the spatial dimension before the video generation model performs inverse diffusion, and the fusion feature sequence is down-sampled from the time sequence, the feature data is compressed, the data amount of the video frame hidden feature in the inverse diffusion process is reduced, the generated video frame number can be improved in the case of limited computing resources, thereby increasing the generated video, and in the model training stage, the features of the sample video are compressed in space and time, the sample data amount is reduced, and the long video can be directly used for model training to improve the video generation quality.

[0070] The technical solutions of the present disclosure will be described in further detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0071] The accompanying drawings, which form a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0072] The present disclosure can be more clearly understood with reference to the following detailed description in conjunction with the accompanying drawings, in which:

[0073] Figure 1 a flowchart of an embodiment of the video generation method of the present disclosure;

[0074] Figure 2 a schematic diagram of the space-time down-sampling process and the space-time up-sampling process in an embodiment of the present disclosure;

[0075] Figure 3 a schematic diagram of the structure of the video generation model in an embodiment of the present disclosure;

[0076] Figure 4 a schematic diagram of the AI video generation process in an embodiment of the present disclosure;

[0077] Figure 5 a flowchart of another embodiment of the video generation method of the present disclosure;

[0078] Figure 6 a schematic diagram of the space-time up-sampling module in the structure encoder in an embodiment of the present disclosure;

[0079] Figure 7 a schematic diagram of the model training process of the video encoder and the video decoder in an embodiment of the present disclosure;

[0080] Figure 8 a schematic diagram of the model training process of the video generation model in an embodiment of the present disclosure;

[0081] Figure 9 a structural schematic diagram of an embodiment of the video generation device of the present disclosure;

[0082] Figure 10 Structure diagram of another embodiment of the video generation device of the present disclosure;

[0083] Figure 11 Structure diagram of one application embodiment of the electronic device of the present disclosure. DETAILED DESCRIPTION

[0084] Various exemplary embodiments of the present disclosure will now be described in detail below with reference to the accompanying drawings. Note that the relative arrangement, numerical expressions, and numerical values of components and steps set forth in these embodiments are not limiting to the scope of the present disclosure unless specifically stated otherwise.

[0085] Those skilled in the art can understand that the terms "first", "second", and the like in the embodiments of the present disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they represent a necessary logical sequence between them.

[0086] It should also be understood that in the embodiments of the present disclosure, "multiple" can mean two or more, and "at least one" can mean one, two, or more.

[0087] It should also be understood that for any component, data, or structure mentioned in the embodiments of the present disclosure, unless specifically limited or given a contrary implication by the context, it can be understood as one or more in general.

[0088] In addition, the term "and / or" in the present disclosure is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present disclosure generally represents an "or" relationship between the front and rear associated objects.

[0089] It should also be understood that the description of various embodiments of the present disclosure focuses on the differences between various embodiments, and the same or similar parts can be referred to each other, and for the sake of brevity, will not be repeated one by one.

[0090] At the same time, it should be understood that, for the convenience of description, the size of each part shown in the drawings is not drawn in accordance with the actual proportional relationship.

[0091] The following description of at least one exemplary embodiment is merely illustrative in nature and does not in any way limit the disclosure and its application or uses.

[0092] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail, but should be considered part of the specification where appropriate.

[0093] It should be noted that like reference numerals and characters refer to like elements throughout the following description and the claims attached hereto. Therefore, once any certain element is defined in one drawing, it is not necessary to discuss it further in connection with other drawings.

[0094] Embodiments of the present disclosure can be applied to terminal devices, computer systems, servers, and other electronic devices, which can operate with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations that can be suitable for use with terminal devices, computer systems, servers, and other electronic devices include, but are not limited to, personal computers, server computers, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputers, mainframe computers, and distributed cloud computing environments that include any of the above systems or the like.

[0095] Terminal devices, computer systems, servers, and other electronic devices can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like, which perform particular tasks or implement particular abstract data types. Computer systems / servers can be practiced in distributed cloud computing environments with remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located in local or remote computer system storage media including memory storage devices.

[0096] Reference is made to Figure 1 which shows a flowchart of a video generation method provided by an example embodiment of the present disclosure. The video generation method of the present embodiment can be implemented by an electronic device deployed with a video generation model and a video decoder. The embodiment includes the following steps:

[0097] At step 101, in response to a video generation instruction, a first frame image and a description text specified by the video generation instruction are obtained, and the description text is used to indicate the video content.

[0098] In a possible implementation, the electronic device has a text-to-video function, and a user can upload a description text through operation, so that the electronic device generates a corresponding AI video by using a video generation model and a video decoder. In order to improve the accuracy of the video content, the user can also upload an image as the first frame image of the target video, so that the video generation model generates subsequent video content on the basis of the first frame image. Optionally, when generating a long video, the user can directly upload the first frame image and the description text, and correspondingly, the video generation instruction directly carries the first frame image and the description text; or the user can also create the first frame image and the description text in advance and store them, and when generating a long video, the user specifies the storage address of the first frame image and the description text, and correspondingly, the video generation instruction includes the storage address of the first frame image and the storage address of the description text. In this step 101, the first frame image and the description text can be obtained from the corresponding storage addresses based on the storage address of the first frame image and the storage address of the description text in the video generation instruction.

[0099] In response to the video generation instruction, the first frame image and the description text are obtained based on the video generation instruction. The description text is used to indicate the video content, for example, "a car is driving forward on a highway, and there is a vehicle merging into the right fork". The first frame image and the description text can be obtained by the electronic device or uploaded by the user's mobile terminal, and the disclosure does not limit the obtaining method of the first frame image and the description text.

[0100] In step 102, the first frame image and the description text are respectively subjected to feature extraction to obtain first frame features and text features.

[0101] Optionally, the electronic device can also deploy an image encoder and a text encoder. The first frame image is subjected to image encoding and feature extraction by the image encoder to obtain the first frame features, and the description text is subjected to text encoding and feature extraction by the text encoder to obtain the text features.

[0102] Illustratively, the text encoder can use a pre-trained neural network model (Contrastive Language-Image Pre-Training, CLIP) to tokenize the text data, and convert it to one-dimensional text features through a linear mapping layer.

[0103] In step 103, the first frame features, the text features, and the random noise are subjected to feature fusion, and the obtained fusion feature sequence is input into the video generation model to generate a target hidden feature sequence corresponding to the video frame sequence in the hidden space by the video generation module.

[0104] The video generation model comprises a space-time downsampling module and an inverse diffusion module. The space-time downsampling module is configured to downsample the fusion features in the spatial dimension and downsample the fusion feature sequence in the time sequence dimension. The inverse diffusion module is configured to perform inverse diffusion processing on the initial feature sequence after downsampling.

[0105] Optionally, the video generation model can be implemented by using a diffusion model (Diffusion Models), for example, a U-Net neural network model. The principle of model inference is to perform inverse diffusion on random noise and gradually remove noise to obtain clear video frame hidden features.

[0106] In a possible implementation, the first frame feature is first copied in the time sequence dimension based on a preset number of video frames, so that the time sequence dimension of the first frame feature is consistent with the hidden feature sequence. For example, if 1000 video frames are to be generated, the first frame feature is copied until the time sequence dimension reaches 1000. Then, the first frame feature is added with Gaussian random noise, and the feature sequence after the noise addition is spliced with the text feature to obtain a fusion feature sequence.

[0107] The fusion feature sequence is input into the video generation model, and the space-time downsampling module can be used to downsample each fusion feature in the fusion feature sequence in the spatial dimension (compress the number of rows and columns of each feature matrix) and in the time sequence dimension (compress the number of frames corresponding to the fusion feature sequence), so as to compress the data amount of the entire fusion feature sequence. The fusion feature sequence after data compression is subjected to inverse diffusion to gradually remove noise and obtain a clear hidden feature sequence. Since the data amount of the fusion feature sequence after downsampling is reduced, the number of video frames can be increased in the case of limited computing resources.

[0108] Correspondingly, the video generation model can also be provided with an upsampling module to upsample the hidden feature sequence obtained by inverse diffusion to obtain a target hidden feature sequence. The space-time dimension of upsampling enhancement is consistent with the space-time dimension of downsampling compression.

[0109] Optionally, the space-time downsampling module can perform one-time space-time downsampling on the fusion feature sequence or multiple times of space-time downsampling.

[0110] In step 104, the target hidden feature sequence is input into a video decoder, and a target video corresponding to the target hidden feature sequence is generated by the video decoder.

[0111] The clear video frames can be obtained by decoding the target hidden feature sequence based on the video decoder, and the target video corresponding to the description text and the first frame image is generated. In a possible implementation, the target video is an AI long video, for example, an AI video with a length of more than 15 minutes.

[0112] In the embodiments of the present disclosure, by adding a space-time downsampling module in the video generation model, the fusion features of each video frame are downsampled from the spatial dimension before the video generation model performs inverse diffusion, and the fusion feature sequence is downsampled from the time sequence, the feature data is compressed, the data amount of the video frame hidden features in the inverse diffusion process is reduced, the generated video frame number can be improved in the case of limited computing resources, thereby increasing the generated video, and in the model training stage, the features of the sample video are compressed in space and time, the sample data amount is reduced, and long videos can be directly used for model training to improve the video generation quality.

[0113] Optionally, based on the embodiments shown in the above Figure 1 The step 103 can include the following steps based on the embodiments shown in the above

[0114] Step 103a, using a space-time downsampling module to perform space-time downsampling on the fusion feature sequence to obtain a downsampled fusion feature sequence.

[0115] Step 103b, using an inverse diffusion module to perform inverse diffusion processing on the downsampled fusion feature sequence to obtain a downsampled hidden feature sequence.

[0116] Step 103c, using a space-time upsampling module of the video generation model to perform space-time upsampling on the downsampled hidden feature sequence to obtain a target hidden feature sequence.

[0117] In a possible implementation, the video generation model includes a space-time downsampling module, an inverse diffusion module and a space-time upsampling module. The space-time downsampling module is used to downsample the fusion feature sequence input into the model in the spatial dimension and the time sequence dimension, and compress the feature data amount; the inverse diffusion module is used to perform at least one inverse diffusion processing on the downsampled fusion feature sequence, and gradually generate a downsampled hidden feature sequence meeting the requirements; and the space-time upsampling module is used to perform space-time upsampling on the downsampled hidden feature sequence to obtain a target hidden feature sequence.

[0118] Optionally, the space-time downsampling module compresses the spatial multiple corresponding to the spatial multiple raised by the space-time upsampling module, and the time sequence multiple corresponding to the time sequence multiple raised. For example, the space-time downsampling module downsampled the fusion feature sequence by 8 times in the spatial dimension and by 4 times in the time sequence dimension, and the space-time upsampling module upsampled by 8 times in the spatial dimension and by 4 times in the time sequence dimension.

[0119] Figure 2A schematic diagram of a process of spatio-temporal down-sampling and a process of spatio-temporal up-sampling of a fused feature sequence is shown. In the diagram, T represents the time dimension of the feature sequence (corresponding to the number of video frames), H represents the height of each feature (the number of matrix rows), W represents the width of each feature (the number of matrix columns), and C1 to C3 represent the number of channels. After the first spatio-temporal down-sampling of the fused feature sequence, the time dimension is halved, and the height and width are each halved. After the second spatio-temporal down-sampling of the fused feature sequence, the time dimension is reduced to 1 / 4 of the original, and the height and width are each reduced to 1 / 4 of the original. After inverse diffusion of the down-sampled fused feature sequence with a spatio-temporal dimension of (T / 4) x C3 x (H / 4) x (W / 4), corresponding multiple spatio-temporal up-sampling is performed to obtain a target hidden feature sequence with a dimension consistent with the fused feature sequence.

[0120] In a possible implementation, the spatio-temporal down-sampling module includes a spatial down-sampling unit and a time sequence down-sampling unit. The spatial down-sampling unit is configured to perform spatial down-sampling on the fused features in the fused feature sequence by using a two-dimensional convolution layer with a preset convolution kernel size and a preset step size. For example, a two-dimensional convolution layer with a convolution kernel size of 3 x 3 and a step size of 2 is used. The time sequence down-sampling unit is configured to perform time sequence down-sampling on the fused feature sequence by using a three-dimensional average pooling layer with a preset convolution kernel size and a preset step size. For example, a three-dimensional average pooling layer with a convolution kernel size of 3 x 1 x 1 and a step size of 2 x 1 x 1 is used.

[0121] Optionally, after the video generation model down-samples the fused feature sequence using the convolution kernel, the video generation model can use an attention network to perform global feature extraction to obtain a down-sampled fused feature sequence. The step 103a can specifically include the following steps:

[0122] The spatio-temporal down-sampling module is used to perform spatio-temporal down-sampling on the fused features to obtain local down-sampled features. The attention network is used to perform self-attention calculation on the local down-sampled features, and cross-attention calculation on the local down-sampled features and the text features, to obtain a down-sampled fused feature sequence.

[0123] Since the local down-sampled features are output by the convolution kernel down-sampling, an attention network can be arranged after the spatio-temporal down-sampling module to perform global feature extraction to obtain a complete down-sampled fused feature sequence. In a possible implementation, the video generation model can include at least one spatio-temporal down-sampling module. When there are multiple spatio-temporal down-sampling modules, the attention network and the spatio-temporal down-sampling module can be arranged alternately, so that global feature extraction is performed after each spatio-temporal down-sampling.

[0124] Optionally, inputting the local down-sampling features and the text features into the attention network can enable the attention network to perform global feature extraction under the constraint of describing the text. After self-attention calculation on the local down-sampling features and cross-attention calculation on the local down-sampling features and the text features by the attention network, the down-sampling fusion feature sequence is obtained. Then, the down-sampling fusion feature sequence and the text features are input into the next spatio-temporal down-sampling module for the next spatio-temporal down-sampling. Illustratively, Figure 3 A structural diagram of a video generation model is shown. The left side is used for spatio-temporal down-sampling and global feature extraction on the fusion feature sequence to obtain a down-sampling fusion feature sequence. Specifically, first, Gaussian random noise is added to the first frame feature, and the obtained feature sequence is input into a spatial down-sampling unit for one-time spatial dimension down-sampling. Then, the feature sequence obtained by spatial down-sampling and the text features are input into a spatio-temporal attention module for self-attention calculation on the local down-sampling features and cross-attention calculation on the local down-sampling features and the text features to obtain a first down-sampling fusion feature sequence. The first down-sampling fusion feature sequence is input into a spatio-temporal down-sampling module for one-time spatio-temporal down-sampling. Then, the feature sequence obtained by spatio-temporal down-sampling and the text features are input into the spatio-temporal attention module for self-attention calculation on the local down-sampling features and cross-attention calculation on the local down-sampling features and the text features to obtain a second down-sampling fusion feature sequence. The above spatio-temporal down-sampling and global feature extraction process is repeated until the last attention network outputs the final obtained down-sampling fusion feature sequence.

[0125] The spatio-temporal down-sampling module and the attention network can be alternately arranged, and a separate spatial down-sampling unit or a time sequence down-sampling unit can be arranged to achieve down-sampling with different multiples of spatial dimension and time sequence dimension. For example, the fusion feature sequence can be subjected to 8 times spatial down-sampling and 4 times time sequence down-sampling. The specific multiples of spatial down-sampling and time sequence down-sampling are not limited in the embodiments of the present disclosure.

[0126] In a possible implementation, the spatio-temporal up-sampling module includes a spatial up-sampling unit and a time sequence up-sampling unit. The spatial up-sampling unit is configured to perform spatial up-sampling on the down-sampling hidden feature sequence by means of spatial nearest neighbor interpolation. The time sequence up-sampling unit is configured to perform time sequence up-sampling on the down-sampling hidden feature sequence by means of time sequence nearest neighbor interpolation.

[0127] Correspondingly, after the spatio-temporal up-sampling module performs spatio-temporal up-sampling on the down-sampling hidden feature sequence, the video generation model can perform global feature extraction on the target hidden feature sequence by using the attention network. The step 103c can specifically include the following steps:

[0128] performing spatio-temporal up-sampling on the down-sampling hidden feature sequence by using the spatio-temporal up-sampling module to obtain a local hidden feature sequence;

[0129] The second attention network is used for self-attention calculation on the local hidden feature, and cross-attention calculation is performed on the local hidden feature and the text feature to obtain a target hidden feature sequence.

[0130] Illustratively, Figure 3 A structural schematic diagram of a video generation model is shown. The right side is used for spatio-temporal upsampling and global feature extraction on the down-sampled hidden feature sequence to obtain a target hidden feature sequence. Specifically, the down-sampled hidden feature sequence obtained after inverse diffusion is first spatially input into a spatio-temporal upsampling module for spatio-temporal upsampling; then the local hidden feature sequence obtained by spatio-temporal upsampling and the text feature are input into a spatio-temporal attention module, self-attention calculation is performed on the local hidden feature, and cross-attention calculation is performed on the local hidden feature and the text feature to obtain a first up-sampled hidden feature sequence; the first up-sampled hidden feature sequence is input into the next spatio-temporal upsampling module for spatio-temporal upsampling; then the local hidden feature obtained by spatio-temporal upsampling and the text feature are input into the spatio-temporal attention module, self-attention calculation is performed on the local hidden feature, and cross-attention calculation is performed on the local hidden feature and the text feature to obtain a second up-sampled hidden feature sequence; the second up-sampled hidden feature sequence is input into a spatial upsampling module (including a spatial upsampling unit) for spatial dimension upsampling; the local hidden feature obtained by spatial upsampling and the text feature are input into the spatio-temporal attention module for global feature extraction to obtain the target hidden feature sequence. Figure 3 The video generation model shown is only an example, which includes one spatial down-sampling module and two spatio-temporal down-sampling modules, one spatial upsampling module and two spatio-temporal upsampling modules. In other possible implementations, the number of modules can be set based on requirements, and the number of modules is not limited in the embodiments of the present disclosure.

[0131] In a possible implementation, the user can also upload structure information, which includes key point coordinates of the target object in each video frame. The method provided by the embodiments of the present disclosure can further include the following steps:

[0132] Obtaining structure information specified by the video generation instruction; and performing feature extraction on the structure information by using a structure encoder to obtain structure features.

[0133] The above step 103 can include the following steps:

[0134] The first frame feature, the text feature, the structure feature, and the random noise are fused, and the obtained fusion feature sequence is input into the video generation model to generate a target hidden feature sequence; and the video decoder is used to generate a target video in which the target object moves in a motion manner indicated by the structure information based on the target hidden feature sequence.

[0135] The structural information includes key point coordinates of the target object in each video frame, and can also include contour information of the target object, and the like. For example, for AI video generation of an automatic driving scene of a car, the structural information can include positions of key points of each vehicle in each video frame in the video. A user can label the positions of the key points of the target object in each frame structure diagram and upload. Alternatively, the user can directly upload the specification structural information, and correspondingly, the video generation instruction directly carries the structural information; or the user can also create the structural information in advance and store, and specify a storage address of the structural information when performing long video generation operation, and correspondingly, the video generation instruction includes the storage address of the structural information.

[0136] According to the embodiments of the present disclosure, by fusing the structured information with the first frame feature, the text feature and the random noise, inputting the video generation model to obtain the target hidden feature, and using the structured information to control the moving manner of the target object in the video, the motion amplitude of the target object can be improved, and the AI video content generated is more realistic.

[0137] Schematically, Figure 4 A target video generation process is shown. As shown in Figure 4 The first frame image is encoded by using the first frame encoder 401 to obtain the first frame feature, and the structural information is encoded by using the structure encoder 402 to obtain the structured information. The structured information is fused and spliced with the first frame feature and the random noise, and is input into the video generation model. The spatial downsampling module a, the spatio-temporal attention module b, the spatio-temporal downsampling module c, the spatio-temporal attention module d, the spatio-temporal downsampling module e, the spatio-temporal attention module f in the video generation model are used for spatio-temporal downsampling and attention calculation, and at least one spatial downsampling and multiple spatio-temporal downsampling are performed. The inverse diffusion processing is performed on the feature sequence after the downsampling, and then the spatio-temporal upsampling module g, the spatio-temporal attention module h, the spatio-temporal upsampling module i, the spatio-temporal attention module j, the spatial upsampling module k, and the spatio-temporal attention module l in the video generation model are used for upsampling and attention calculation to obtain the target hidden feature sequence. The target video is obtained by decoding the target hidden feature sequence by using the video decoder 404. Alternatively, the first frame encoder 401 and the structure encoder 402 are also provided with spatio-temporal downsampling modules, which are respectively used for spatio-temporal downsampling of the first frame image and the structural information, and the downsampling is performed on the first frame image and the structural information before being input into the video generation model. Figure 4 It can be seen that the feature dimension of the target hidden feature sequence in time sequence and space is less than the feature dimension of the structural information, and the feature dimension in space is also less than the dimension of the first frame image feature in space. Correspondingly, the spatio-temporal upsampling module is arranged in the video decoder 404, which is used for spatio-temporal upsampling of the target hidden feature sequence, and the target video is obtained after improving the time sequence and spatial feature dimension of the feature.

[0138] In a possible implementation, the model training is performed first before the target video is generated by using the model inference. The model training mainly includes two parts, one part is to train the video decoder, and the other part is to train the video generation model. The method provided by the embodiment of the present disclosure is described below with reference to Figure 5 Before step 101, the following step can be included:

[0139] Step 501, a model training framework is constructed.

[0140] The model training framework includes a video encoder, a video generation model, a video decoder, a first frame encoder and a text encoder. The video encoder and the video generation model each include a spatial-temporal up-sampling module. The video generation model and the video decoder each include a spatial-temporal down-sampling module.

[0141] In a possible implementation, the video encoder and the video decoder are trained together, and the video decoder capable of generating a long video from the latent feature can be obtained. Meanwhile, the video encoder can be used to encode the sample video when the video generation model is trained, so as to provide the sample latent feature for the training of the video generation model. The first frame encoder is used to encode the first frame image of the video to obtain the first frame feature. In order to make the first frame feature and the latent feature of the sample video in the same latent space, the first frame encoder can be implemented by reusing the video encoder. The text encoder is used to encode the sample description text, and a pre-trained model can be directly used.

[0142] Optionally, the model training framework can further include a structure encoder. The structure encoder is used to encode the sample structure information to obtain a sample structure feature, so that the sample structure feature, the sample latent feature, the sample first frame feature and the sample text feature are input into the video generation model, and the video generation model learns to control the motion amplitude of the target object based on the structure information. The spatial-temporal down-sampling module is arranged in the structure encoder and the first frame encoder, and is respectively used for spatial-temporal down-sampling of the structure information and the first frame image. The spatial-temporal dimensions of the sample structure feature, the sample first frame feature and the sample latent feature encoded by the video encoder are consistent.

[0143] Figure 6 A structure diagram of a spatial-temporal down-sampling module in a structure encoder is shown. The module includes three two-dimensional convolution layers with a convolution kernel size of 3x3 and a step size of 2, which are used for spatial down-sampling of the sample structure feature. The module also includes two three-dimensional average pooling layers with a convolution kernel size of 3x1x1 and a step size of 2x1x1, which are used for time sequence down-sampling of the sample structure feature. Figure 6 The structure encoder shown is only used as an example for description. In other possible implementations, the convolution kernel size, the step size and the number of convolution layers of the structure encoder can be set according to actual needs, and the embodiment of the present disclosure does not limit the same.

[0144] Step 502: Fix the parameters of the video generation model and train the video encoder and video decoder using sample videos.

[0145] The first stage of model training involves fixing the parameters of the video generation model and training the video encoder and decoder using sample videos. The video encoder includes a spatiotemporal downsampling module, and the video decoder includes a spatiotemporal upsampling module. The video encoder uses the spatiotemporal downsampling module to downsample the encoded sample latent feature sequences in both spatial and temporal dimensions, reducing the amount of data in the sample latent feature sequences. The specific downsampling method of the spatiotemporal downsampling module and the upsampling method of the spatiotemporal upsampling module can be found in the downsampling and upsampling methods of the video generation model in the above embodiments.

[0146] Indicative, such as Figure 7 As shown, the sample video 701 can be encoded and spatiotemporally downsampled using the video encoder 702 to obtain the downsampled sample latent feature sequence 703. It can be seen that the latent feature sequence 703 has a smaller temporal and spatial dimension than the initial feature sequence of the sample video. The video decoder 704 performs upsampling processing on the latent feature sequence 703 and decodes it to generate a reconstructed video 705. The video encoder 702 and video decoder 703 are iteratively trained using the sample video 701 and the reconstructed video 705 until a preset training completion condition is met, at which point training is complete. The preset training completion condition may include, but is not limited to, at least one of the following: the number of iterative training iterations reaches a preset number of training iterations (e.g., 1000 times); the function value of the first loss function calculated based on the sample video and the reconstructed video is less than a preset threshold, etc. This embodiment of the present disclosure does not limit this.

[0147] Step 502 above may include the following steps:

[0148] Step 502a: Use a video encoder to perform video encoding and spatiotemporal downsampling on the sample latent feature sequence to obtain the mean and standard deviation of the sample latent feature sequence.

[0149] Step 502b: Perform parameter renormalization based on the mean and standard deviation to obtain the latent feature sequence of the sample.

[0150] Step 502c: Upsample the latent feature sequence of the sample and perform video decoding using the input video decoder to obtain the reconstructed video.

[0151] Step 502d: Update the parameters of the video encoder and video decoder based on the sample video and the reconstructed video.

[0152] Specifically, the first loss function can be used to calculate its value based on the sample video and the reconstructed video, thus obtaining the first loss function value. The parameters of the video encoder and video decoder are then updated based on this first loss function value. The first loss function is:

[0153] L = L rec +0.000001·L kl

[0154]

[0155] L kl = -0.5(logσ) 2 +1-μ 2 -σ 2 )

[0156] Where σ is the standard deviation of the sample latent feature sequence, and μ is the mean of the sample latent feature sequence. Let y be the output of the video decoder and y be the input of the video encoder. The mean and standard deviation of the sample latent feature sequence can be obtained through the video encoder, and then the parameters can be renormalized to obtain the encoded sample latent feature sequence. Illustratively, the sample latent feature z = μ + σ·n, where n is random Gaussian noise.

[0157] The first loss function L is derived from the KL loss function L. kl and reconstruction loss function L rec It consists of two parts: the KL loss function is calculated based on the mean and standard deviation of the video encoder output, and the reconstruction loss function is calculated based on the feature differences between the reconstructed video and the sample video.

[0158] Step 503: In response to the completion of training of the video encoder and video decoder, fix the parameters of the video encoder and video decoder, and train the video generation model using sample videos.

[0159] Indicative, such as Figure 8 As shown, after the video encoder 803 and video decoder are trained, the video encoder 803 can be used to generate the latent feature sequence of the sample video (from...). Figure 8 It can be seen that the temporal and spatial dimensions of the sample latent feature sequence are smaller than those of the initial feature sequence of the sample video. The noisy sample latent feature sequence and other information (sample first-frame features generated by the first-frame encoder 801 based on the sample first-frame image, and sample structural features generated by the structure encoder 802 based on the sample structural information) are input into the video generation model for spatiotemporal downsampling, inverse diffusion, and spatiotemporal upsampling to obtain the predicted noise output by the video generation model. The model is then iteratively updated based on the predicted noise and the sample noise. Step 503 above may include the following steps:

[0160] Step 503a, using the trained video encoder to perform video encoding processing and spatio-temporal downsampling on the sample video to obtain a sample hidden feature sequence.

[0161] Step 503b, adding noise to the sample hidden feature sequence using sample noise, and fusing the sample hidden feature sequence after adding noise with a sample first-frame feature and a sample text feature to obtain a sample fusion feature sequence, the sample first-frame feature being obtained by performing feature extraction and spatio-temporal downsampling on a first-frame image of the sample video through a first-frame encoder, and the sample text feature being obtained by performing feature extraction on a sample description text of the sample video through a text encoder.

[0162] Step 503c, performing spatio-temporal downsampling and inverse diffusion processing on the sample fusion feature sequence using a video generation model to obtain predicted noise.

[0163] Step 503d, updating parameters of the video generation model based on the sample noise and the predicted noise.

[0164] Specifically, a second loss function value can be obtained by calculating a function value of the second loss function based on the sample noise and the predicted noise, and updating the parameters of the video generation model based on the second loss function value. The second loss function is:

[0165]

[0166] wherein, ε t is the sample noise added for the t-th time, x t is the sample hidden feature sequence after adding noise for the t-th time, c is a control condition including the sample first-frame feature and the sample text feature, ε θ (x t , t, c) is the predicted noise obtained by the video generation model for predicting the sample noise added for the t-th time.

[0167] According to the embodiments of the present disclosure, the spatio-temporal downsampling is performed on the sample hidden feature sequence of the sample video by the video encoder to reduce the data amount of the sample hidden feature sequence, so that the model training of the video generation model can be directly performed by using the long video, and the video generated by the video generation model in the model inference stage is longer and more coherent and natural, thereby improving the video quality.

[0168] For reference Figure 9 which shows a structural block diagram of a video generation apparatus provided by one exemplary embodiment of the present disclosure. The video generation apparatus provided by the embodiment includes:

[0169] The information acquisition module 901 is configured to acquire a first-frame image and a description text specified by a video generation instruction in response to the video generation instruction, the description text being used to indicate video content.

[0170] The feature extraction module 902 is configured to perform feature extraction on the first frame image and the description text respectively to obtain first frame features and text features.

[0171] The feature generation module 903 is configured to perform feature fusion on the first frame features, the text features, and random noise, input the obtained fusion feature sequence into a video generation model, and generate a target hidden feature sequence corresponding to the video frame sequence in the hidden space through the video generation model. The video generation model includes a space-time downsampling module and an inverse diffusion module. The space-time downsampling module is configured to perform downsampling on the fusion features of each video frame in the spatial dimension and perform downsampling on the fusion feature sequence in the time sequence dimension. The inverse diffusion module is configured to perform inverse diffusion processing based on the initial feature sequence after downsampling.

[0172] The video generation module 904 is configured to input the target hidden feature sequence into a video decoder and generate a target video corresponding to the target hidden feature sequence through the video decoder.

[0173] Optionally, in a possible implementation, the feature generation module 903 of the embodiment of the present disclosure is further configured to:

[0174] perform space-time downsampling on the fusion feature sequence by using the space-time downsampling module to obtain a down-sampled fusion feature sequence;

[0175] perform inverse diffusion processing on the down-sampled fusion feature sequence by using the inverse diffusion module to obtain a down-sampled hidden feature sequence;

[0176] perform space-time upsampling on the down-sampled hidden feature sequence by using a space-time upsampling module of the video generation model to obtain a target hidden feature sequence.

[0177] Optionally, in a possible implementation, the feature generation module 903 of the embodiment of the present disclosure is further configured to:

[0178] perform space-time downsampling on the fusion feature by using the space-time downsampling module to obtain a local down-sampled feature;

[0179] perform self-attention calculation on the local down-sampled feature by using the first attention network, and perform cross-attention calculation on the local down-sampled feature and the text feature to obtain a down-sampled fusion feature sequence;

[0180] perform space-time upsampling on the down-sampled hidden feature sequence by using the space-time upsampling module to obtain a local hidden feature sequence;

[0181] perform self-attention calculation on the local hidden feature by using the second attention network, and perform cross-attention calculation on the local hidden feature and the text feature to obtain a target hidden feature sequence.

[0182] Optionally, in one possible implementation, the spatiotemporal downsampling module of the video generation model of this disclosure includes a spatial downsampling unit and a temporal downsampling unit, and the spatiotemporal upsampling module includes a spatial upsampling unit and a temporal upsampling unit.

[0183] The spatial downsampling unit is used to spatially downsample the fused features in the fused feature sequence through a two-dimensional convolutional layer with a preset kernel size and a preset stride.

[0184] The temporal downsampling unit is used to perform temporal downsampling on the fused feature sequence through a three-dimensional average pooling layer with a preset convolutional kernel size and a preset stride.

[0185] The spatial upsampling unit is used to spatially upsample the downsampled latent feature sequence by spatial nearest neighbor interpolation.

[0186] The temporal upsampling unit is used to perform temporal upsampling on the downsampled latent feature sequence by means of temporal nearest neighbor interpolation.

[0187] Optionally, in one possible implementation, the information acquisition module 901 is further configured to acquire structural information specified by the video generation instruction, the structural information including the key point coordinates of the target object in each video frame;

[0188] The feature extraction module 902 is also used to extract features from the structural information using a structural encoder to obtain structural features;

[0189] The feature generation module 903 is also used to perform feature fusion on the first frame features, text features, structural features and random noise, and input the obtained fused feature sequence into the video generation model to generate the target latent feature sequence; the video decoder is used to generate the target video based on the target latent feature sequence, wherein the target object moves in accordance with the motion mode indicated by the structural information.

[0190] Optional, such as Figure 10 The diagram illustrates a structural block diagram of a video generation apparatus provided in another exemplary embodiment of this disclosure. In one possible implementation, the video generation apparatus of this embodiment further includes:

[0191] Framework construction module 1001 constructs the model training framework, which includes a video encoder, a video generation model, a video decoder, a first-frame encoder, and a text encoder. The video encoder and video generation model each include a spatiotemporal upsampling module, and the video generation model and video decoder each include a spatiotemporal downsampling module.

[0192] The first training module 1002 fixes the parameters of the video generation model and trains the video encoder and video decoder using sample videos.

[0193] The second training module 1003, in response to the training of the video encoder and the video decoder being completed, fixes the parameters of the video encoder and the video decoder, and performs model training on the video generation model by using a sample video.

[0194] Optionally, in a possible implementation, the first training module 1002 of the embodiment of the present disclosure is further configured to:

[0195] perform video encoding processing and spatio-temporal downsampling on the sample video by using the video encoder to obtain the mean and the standard deviation of the sample hidden feature sequence;

[0196] perform parameter renormalization based on the mean and the standard deviation to obtain the sample hidden feature sequence;

[0197] perform upsampling and video decoding processing on the sample hidden feature sequence by using the video decoder to obtain a reconstructed video;

[0198] update the parameters of the video encoder and the video decoder based on the sample video and the reconstructed video.

[0199] Optionally, in a possible implementation, the first training module 1002 of the embodiment of the present disclosure is further configured to:

[0200] calculate a function value of the first loss function based on the sample video and the reconstructed video by using the first loss function to obtain a first loss function value; and update the parameters of the video encoder and the video decoder based on the first loss function value;

[0201] The first loss function is as follows:

[0202] L = L rec + 0.000001 · L kl

[0203]

[0204] L kl = -0.5 (log σ 2 + 1 - μ 2 - σ 2 )

[0205] wherein σ is the standard deviation of the sample hidden feature sequence, μ is the mean of the sample hidden feature sequence, is the output of the video decoder, and y is the input of the video encoder.

[0206] Optionally, in a possible implementation, the second training module 1003 of the embodiment of the present disclosure is further configured to:

[0207] perform video encoding processing and spatio-temporal downsampling on the sample video by using the trained video encoder to obtain the sample hidden feature sequence;

[0208] The sample hidden feature sequence is added with sample noise, and the sample hidden feature sequence after adding the sample noise is fused with a sample first frame feature and a sample text feature to obtain a sample fusion feature sequence, the sample first frame feature is obtained by performing feature extraction and spatio-temporal down-sampling on a first frame image of the sample video through a first frame encoder, and the sample text feature is obtained by performing feature extraction on a sample description text of the sample video through a text encoder;

[0209] The sample fusion feature sequence is subjected to spatio-temporal down-sampling and inverse diffusion processing by using a video generation model to obtain predicted noise;

[0210] Based on the sample noise and the predicted noise, the parameters of the video generation model are updated.

[0211] Optionally, in a possible implementation, the second training module 1003 of the embodiment of the present disclosure is further configured to:

[0212] The second loss function is used to calculate a function value of the second loss function based on the sample noise and the predicted noise to obtain a second loss function value;

[0213] The parameters of the video generation model are updated based on the second loss function value;

[0214] The second loss function is as follows:

[0215]

[0216] wherein, ε t is the sample noise added for the t th time, x t is the sample hidden feature sequence after adding the sample noise for the t th time, c is a control condition including the sample first frame feature and the sample text feature, and ε θ (x t , t, c) is the predicted noise obtained by the video generation model for predicting the sample noise added for the t th time.

[0217] Each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same, similar or corresponding parts of each embodiment can be mutually referred to. Since the system, method, device and equipment embodiments are basically corresponding, the relevant parts can be mutually referred to. The system, method, device and equipment of the embodiment of the present disclosure are also mutually corresponding in specific implementation and beneficial technical effects, and the relevant contents can be mutually referred to.

[0218] In addition, the embodiment of the present disclosure further provides an electronic device, comprising:

[0219] a memory configured to store a computer program;

[0220] The processor is configured to execute a computer program stored in the memory, and the computer program, when executed, implements the video generation method of any one of the embodiments of the present disclosure.

[0221] Figure 11 FIG. 1 shows a schematic diagram of an electronic device according to an embodiment of the present disclosure. Figure 11 The electronic device can be any one or both of the first device and the second device, or a standalone device independent of the first device and the second device, which can communicate with the first device and the second device to receive the collected input signals therefrom.

[0222] As shown in FIG. 1, the electronic device includes one or more processors and a memory. Figure 11

[0223] The processor can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions.

[0224] The memory can include one or more computer program products, which can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM), cache, and / or the like. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage media, and the processor can execute the program instructions to implement the video generation method of the embodiments of the present disclosure described above and / or other desired functions.

[0225] In one example, the electronic device can further include input and output devices, which are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0226] In addition, the input device can further include, for example, a keyboard, a mouse, and / or the like.

[0227] The output device can output various information to the outside, including the determined distance information, direction information, and / or the like. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and / or the like.

[0228] Of course, in order to simplify, Figure 11 ​Only some of the components of the electronic device related to the present disclosure are shown in the figure, and components such as a bus, an input / output interface, and the like are omitted. In addition, the electronic device can further include any other appropriate components according to a specific application.

[0229] In addition to the above-mentioned method and device, an embodiment of the present disclosure can also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the video generation method according to various embodiments of the present disclosure described in the above parts of the specification.

[0230] The computer program product can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, and the like, and conventional procedural programming languages, such as the "C" programming language, or the like. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.

[0231] In addition, an embodiment of the present disclosure can also be a computer readable storage medium, which stores computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the video generation method according to various embodiments of the present disclosure described in the above parts of the specification.

[0232] The computer readable storage medium can take any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of readable storage medium include: an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination of the above.

[0233] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction-related hardware, and the aforementioned program can be stored in a computer readable storage medium, which, when executed, performs steps including the above-mentioned method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disc or optical disc, and various media that can store program codes.

[0234] The above generally describes the basic principles of the present disclosure in conjunction with specific embodiments, but it should be noted that the advantages, benefits, effects and the like mentioned in the present disclosure are only examples and are not limiting, and these advantages, benefits, effects and the like cannot be considered as necessary for each embodiment of the present disclosure. In addition, the above specific details of the disclosure are only for the purpose of example and for the purpose of understanding, and are not limiting, and the above details do not limit the present disclosure to be necessarily implemented with the above specific details.

[0235] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between each embodiment can be mutually referred to. For system embodiments, since they basically correspond to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0236] The block diagrams of the devices, apparatuses, equipment, systems involved in the present disclosure are only illustrative examples and are not intended to require or imply the connection, arrangement, configuration as shown in the block diagram. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have" and the like are open-ended words, which mean "including but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0237] The methods and apparatuses of the present disclosure can be implemented in many ways. For example, the methods and apparatuses of the present disclosure can be implemented by software, hardware, firmware, or any combination of software, hardware, firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present disclosure are not limited to the above specifically described order, unless otherwise specifically described. In addition, in some embodiments, the present disclosure can also be implemented as programs recorded in recording media, which include machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers the recording media storing the programs for executing the method according to the present disclosure.

[0238] It should also be noted that in the apparatuses, equipment and methods of the present disclosure, each component or step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the present disclosure.

[0239] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0240] The above description has been presented to enable any person skilled in the art to make or use the disclosure. Furthermore, the purpose of the above description is not intended to limit the embodiments of the present disclosure to the form disclosed herein. Although various example aspects and embodiments have been discussed above, those of ordinary skill in the art will appreciate a variety of modifications, alternatives, permutations, additions, and sub-combinations of the described aspects and embodiments.

Claims

1. A method of video generation, the method comprising: The method comprises: in response to a video generation instruction, obtaining a first frame image and description text specified by the video generation instruction, the description text being used to indicate video content; performing feature extraction on the first frame image and the description text respectively to obtain first frame features and text features; performing feature fusion on the first frame features, the text features, and random noise, inputting a fusion feature sequence obtained into a video generation model, and generating a target hidden feature sequence corresponding to a video frame sequence in a hidden space through the video generation model; wherein the video generation model comprises a space-time downsampling module and an inverse diffusion module, the space-time downsampling module is used to downsample fusion features of each video frame in a spatial dimension and downsample the fusion feature sequence in a time sequence dimension, and the inverse diffusion module is used to perform inverse diffusion processing based on an initial feature sequence after downsampling; inputting the target hidden feature sequence into a video decoder to generate a target video corresponding to the target hidden feature sequence through the video decoder; wherein, before the step of in response to a video generation instruction, obtaining a first frame image and description text specified by the video generation instruction, the method further comprises: constructing a model training framework, the model training framework comprising a video encoder, the video generation model, the video decoder, a first frame encoder, and a text encoder, the video encoder and the video generation model each comprising the space-time downsampling module, and the video generation model and the video decoder each comprising a space-time upsampling module; fixing parameters of the video generation model, and performing model training on the video encoder and the video decoder using sample videos; in response to the video encoder and the video decoder being trained, fixing parameters of the video encoder and the video decoder, and performing model training on the video generation model using the sample videos.

2. The method of claim 1, wherein, the step of generating a target hidden feature sequence corresponding to a video frame sequence in a hidden space through the video generation model comprises: performing space-time downsampling on the fusion feature sequence using the space-time downsampling module to obtain a downsampled fusion feature sequence; performing inverse diffusion processing on the downsampled fusion feature sequence using the inverse diffusion module to obtain a downsampled hidden feature sequence; performing space-time upsampling on the downsampled hidden feature sequence using a space-time upsampling module of the video generation model to obtain the target hidden feature sequence.

3. The method of claim 2, wherein, the step of performing space-time downsampling on the fusion feature sequence using the space-time downsampling module to obtain a downsampled fusion feature sequence comprises: performing space-time downsampling on the fusion feature using the space-time downsampling module to obtain a local downsampled feature; performing self-attention calculation on the local downsampled feature using a first attention network, and performing cross-attention calculation on the local downsampled feature and the text feature to obtain the downsampled fusion feature sequence; the step of performing space-time upsampling on the downsampled hidden feature sequence using a space-time upsampling module of the video generation model to obtain the target hidden feature sequence comprises: The spatio-temporal up-sampling module is configured to perform spatio-temporal up-sampling on the down-sampled hidden feature sequence to obtain a local hidden feature sequence. The second attention network is configured to perform self-attention calculation on the local hidden feature and cross-attention calculation on the local hidden feature and the text feature to obtain the target hidden feature sequence.

4. The method of claim 3, wherein, The spatio-temporal down-sampling module comprises a spatial down-sampling unit and a temporal down-sampling unit, and the spatio-temporal up-sampling module comprises a spatial up-sampling unit and a temporal up-sampling unit. The spatial down-sampling unit is configured to perform spatial down-sampling on the fusion feature in the fusion feature sequence by a two-dimensional convolution layer with a preset convolution kernel size and a preset step length. The temporal down-sampling unit is configured to perform temporal down-sampling on the fusion feature sequence by a three-dimensional average pooling layer with a preset convolution kernel size and a preset step length. The spatial up-sampling unit is configured to perform spatial up-sampling on the down-sampled hidden feature sequence by a spatial nearest neighbor interpolation method. The temporal up-sampling unit is configured to perform temporal up-sampling on the down-sampled hidden feature sequence by a temporal nearest neighbor interpolation method.

5. The method according to any one of claims 1 to 4, characterized in that, Further comprising: obtaining structure information specified by the video generation instruction, the structure information comprising key point coordinates of a target object in each video frame; extracting features from the structure information by a structure encoder to obtain structure features; The feature fusion of the first frame feature, the text feature and the random noise is performed, and the obtained fusion feature sequence is input into a video generation model, and the target hidden feature sequence corresponding to the video frame sequence in the hidden space generated by the video generation model comprises: The feature fusion of the first frame feature, the text feature, the structure feature and the random noise is performed, and the obtained fusion feature sequence is input into the video generation model, and the target hidden feature sequence is generated by the video generation model; The video decoder is configured to generate the target video in which the target object moves in a motion manner indicated by the structure information based on the target hidden feature sequence.

6. The method of claim 1, wherein, The model training of the video encoder and the video decoder by the sample video comprises: performing video encoding processing and spatio-temporal down-sampling on the sample video by the video encoder to obtain a mean value and a standard deviation of a sample hidden feature sequence; performing parameter renormalization based on the mean value and the standard deviation to obtain the sample hidden feature sequence; performing up-sampling and video decoding processing on the sample hidden feature sequence by the video decoder to obtain a reconstructed video; updating parameters of the video encoder and the video decoder based on the sample video and the reconstructed video.

7. The method of claim 6, wherein, The parameter updating of the video encoder and the video decoder based on the sample video and the reconstructed video comprises: calculating a function value of the first loss function based on the sample video and the reconstructed video by a first loss function to obtain a first loss function value; updating parameters of the video encoder and the video decoder based on the first loss function value; The first loss function is: L = L rec + 0.000001 • L kl L kl = -0.5(logσ 2 + 1 - μ 2 - σ 2 ) wherein σ is a standard deviation of the sample latent feature sequence, μ is a mean of the sample latent feature sequence, is an output of the video decoder, y is an input of the video encoder.

8. The method of claim 1, wherein, The model training of the video generation model by the sample video comprises: The sample video is video coded and spatio-temporally down-sampled by using the trained video encoder to obtain a sample hidden feature sequence; The sample hidden feature sequence is added with sample noise, and the sample hidden feature sequence added with the sample noise is fused with a sample first frame feature and a sample text feature to obtain a sample fusion feature sequence, the sample first frame feature being obtained by performing feature extraction and spatio-temporal down-sampling on a first frame image of the sample video by using a first frame encoder, and the sample text feature being obtained by performing feature extraction on a sample description text of the sample video by using a text encoder; The sample fusion feature sequence is spatio-temporally down-sampled and inverse diffused by using the video generation model to obtain predicted noise; Parameters of the video generation model are updated based on the sample noise and the predicted noise.

9. The method of claim 8, wherein, The updating of the parameters of the video generation model based on the sample noise and the predicted noise comprises: A second loss function value is obtained by calculating a function value of the second loss function based on the sample noise and the predicted noise by using the second loss function; The parameters of the video generation model are updated based on the second loss function value; The second loss function is: wherein ε t is the sample noise added for the t-th time, x t is the sample hidden feature sequence after the t-th time of adding noise, c is a control condition including the sample first frame feature and the sample text feature, ε θ (x t ,t,c) is the prediction noise obtained by the video generation model for the t-th time of adding the sample noise.

10. A video generating apparatus characterized by comprising: The device comprises: An information acquisition module is configured to acquire a first frame image and a description text specified by a video generation instruction in response to the video generation instruction, the description text being used to indicate video content; A feature extraction module is configured to perform feature extraction on the first frame image and the description text respectively to obtain a first frame feature and a text feature; A feature generation module is configured to fuse the first frame feature, the text feature and random noise to obtain a fusion feature sequence, and input the fusion feature sequence into a video generation model to generate a target hidden feature sequence corresponding to a video frame sequence in a hidden space, wherein the video generation model comprises a spatio-temporal down-sampling module and an inverse diffusion module, the spatio-temporal down-sampling module is configured to down-sample the fusion feature of each video frame in a spatial dimension and down-sample the fusion feature sequence in a time sequence dimension, and the inverse diffusion module is configured to perform inverse diffusion processing based on an initial feature sequence after down-sampling; A video generation module is configured to input the target hidden feature sequence into a video decoder to generate a target video corresponding to the target hidden feature sequence. The device further comprises: A framework construction module is configured to construct a model training framework, the model training framework comprising a video encoder, the video generation model, the video decoder, a first frame encoder and a text encoder, the video encoder and the video generation model each comprising the spatio-temporal down-sampling module, and the video generation model and the video decoder each comprising a spatio-temporal up-sampling module; A first training module is configured to fix parameters of the video generation model, and perform model training on the video encoder and the video decoder by using a sample video; A second training module is configured to fix parameters of the video encoder and the video decoder in response to completion of training of the video encoder and the video decoder, and perform model training on the video generation model by using the sample video.

11. An electronic device, comprising: The computer program product comprises: a memory for storing a computer program; a processor for executing the computer program stored in the memory, and the computer program, when executed, implements the method of any one of claims 1-9.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the method of any one of claims 1-9.

13. A computer program product comprising computer program instructions, characterised in that, The computer program instructions, when executed by the processor, implement the method of any one of claims 1-9.

Images