Video generation method, device and cluster

Through the pre-trained diffusion model and denoising and feature update of noise samples, the problem of difficult video interframe consistency in text-to-video technology is solved, and efficient and low-cost video generation is achieved.

CN118945442APending Publication Date: 2024-11-12HUAWEI CLOUD COMPUTING TECHNOLOGIES CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310912526.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-12
Filing Date
2023-07-24
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing text-to-video technologies have difficulties in generating video interframe consistency, requiring a large amount of computing resources and difficult to control the effects.

Method used

The pre-trained diffusion model is adopted to obtain text description information and structural information, and use noise samples to perform denoising and feature updates to achieve high inter-frame consistency video generation without retraining the model.

Benefits of technology

Without training the model, generating videos with high consistency between frames reduces the computational complexity and computing power requirements and improves the quality and efficiency of video generation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118945442A_ABST
    Figure CN118945442A_ABST
Patent Text Reader

Abstract

The invention discloses a video generation method, device and cluster. The method comprises the following steps: acquiring text description information and multiple pieces of structure information of a target video, wherein the multiple pieces of structure information are in one-to-one correspondence with multiple noise samples; in the first time step, obtaining a first noise sample in the plurality of noise samples, the first noise sample being used for generating a first video frame in the target video; removing the noise of the first noise sample based on the text description information and the structure information corresponding to the first noise sample by using a pre-trained diffusion model to obtain a first intermediate sample corresponding to the first noise sample; and updating features of a first intermediate sample corresponding to the first noise sample to obtain a first output sample corresponding to the first noise sample; wherein the first output sample of the first noise sample is used as a noise sample of a next time step of the first time step, or the first video frame. According to the method, the video with high inter-frame consistency can be generated without training a model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video generation method, device and cluster. Background Art

[0002] Text to video technology is a technology that uses artificial intelligence (AI) to generate videos whose content matches the text description. Users can use text information to generate the videos they want through text to video technology, which can greatly save the time of video creation and editing.

[0003] In order to ensure consistency between video frames in the generated video, the model currently used in text-to-video technology is retrained using video samples based on the text-to-image generation model. This consumes a lot of computing resources, and the effect of retraining is difficult to control, resulting in difficult to control the video generation effect. Summary of the invention

[0004] The embodiments of the present application provide a video generation method, device and cluster, which can generate videos with high inter-frame consistency without the need for training models.

[0005] In a first aspect, a video generation method is provided, the method comprising: obtaining text description information and multiple structural information of a target video, the multiple structural information corresponding to multiple noise samples one-to-one; wherein each noise sample is used to generate a video frame in the target video, and the structural information corresponding to the noise sample is used to indicate the structure of an object in the video frame; in a first time step, obtaining a first noise sample from multiple noise samples, the first noise sample being used to generate a first video frame in the target video; using a pre-trained diffusion model, based on the text description information and the structural information corresponding to the first noise sample, removing noise from the first noise sample to obtain a first intermediate sample corresponding to the first noise sample; and updating a feature of the first intermediate sample corresponding to the first noise sample to obtain a first output sample corresponding to the first noise sample; wherein the first output sample of the first noise sample is used as a noise sample of the next time step of the first time step, or a first video frame; wherein updating the feature of the first intermediate sample corresponding to the first noise sample comprises: based on an attention mechanism, fusing the feature of the first intermediate sample corresponding to the first noise sample with the feature of the first intermediate sample corresponding to the first noise sample; wherein the other noise samples are used to generate other video frames in the target video except the first video frame.

[0006] The pre-trained diffusion model refers to a pre-trained diffusion model, that is, an existing diffusion model that is not trained for implementing the method. The diffusion model may specifically be a text-image diffusion model, that is, a model for generating a target image based on text description information and structural information.

[0007] In this method, a pre-trained diffusion model can be used to remove noise from noise samples and update relevant features of noise samples to achieve the generation of target video. Specifically, the denoising parameter set of the diffusion model can be used to calculate the noise sample to remove the noise of the noise sample; and the attention parameter of the diffusion model can be used to fuse the relevant features of other noise samples to the first noise sample, so that in the process of generating the first video frame based on the first noise sample, the relevant features of other noise samples are fused into the relevant features of the first noise sample, so that the consistency between the first video frame and the video frames generated by other noise samples can be improved.

[0008] In short, this method can generate a target video with high inter-frame consistency without retraining the diffusion model or other models and training a new model.

[0009] In a possible implementation, the first intermediate sample corresponding to the first noise sample includes a first region and a second region, and updating the features of the first intermediate sample corresponding to the first noise sample further includes: based on an attention mechanism, fusing features of the second region with features of the first region, and fusing features of the first region with features of the second region.

[0010] In this implementation, while fusing the relevant features of other noise samples with the relevant features of the noise sample, it is also possible to fuse the features of one area of ​​the noise sample with the features of another area of ​​the noise sample, thereby improving the intra-frame consistency of the video frame while improving the inter-frame consistency of the video frame.

[0011] In a possible implementation, the method further includes: updating the first output sample corresponding to the first noise sample based on the first output sample corresponding to the second noise sample and the first output sample corresponding to the third noise sample; wherein the second noise sample is used to generate a second video frame in the target video, the third noise sample is used to generate a third video frame in the target video, and the second video frame, the first video frame and the third video frame are adjacent in sequence in the target video.

[0012] In this implementation, in the process of generating video frames based on noise samples, the features of the output samples corresponding to the noise samples on both sides of the current noise sample are used to update the output samples of the current noise sample. Thus, based on one of the two adjacent noise samples, the other noise sample can be smoothed, which can further improve the consistency between the two adjacent video frames.

[0013] In a possible implementation, the method also includes: in the second time step, using a diffusion model, based on the text description information and the structural information corresponding to the first noise sample, removing the noise of the first output sample corresponding to the first noise sample to obtain the second intermediate sample corresponding to the first noise sample; and updating the features of the second intermediate sample corresponding to the first noise sample to obtain the second output sample corresponding to the first noise sample; based on the second output sample corresponding to the first noise sample and the second output sample corresponding to the fourth noise sample among the multiple noise samples, updating the second output sample corresponding to the third noise sample; wherein the fourth noise sample is used to generate a fourth video frame in the target video, the first video frame, the third video frame and the fourth video frame are adjacent in sequence in the target video, and the second time step is the next time step of the first time step.

[0014] In this implementation, in the second time step, based on smoothing one of two adjacent noise samples in the first time step, the other noise sample is smoothed in the second time step, which can further improve the consistency between two adjacent video frames.

[0015] In a possible implementation, the target video includes a plurality of segments connected in sequence, each segment includes a key frame and a non-key frame, and the first video frame and the other video frames are key frames of different segments respectively; after updating the features of the first intermediate sample corresponding to the first noise sample to obtain the first output sample corresponding to the first noise sample, the method further includes: based on an attention mechanism, fusing the features of the first output sample corresponding to the first noise sample and the features of the first intermediate sample corresponding to the fifth noise sample with the features of the first intermediate sample corresponding to the fifth noise sample to obtain the first output sample of the fifth noise sample; wherein the fifth noise sample and the sixth noise sample are respectively used to generate different non-key frames in the segment where the first video frame is located.

[0016] In this implementation, the consistency between different segments is achieved by integrating the features of noise samples corresponding to the key frames of other segments into the features of noise samples corresponding to the key frames of the segments. The features of noise samples corresponding to other frames in the segment are integrated into the features of noise samples corresponding to the non-key frames of the segment, and the other frames include the key frames of the segment, so as to ensure the consistency between video frames in the segment while ensuring the consistency between segments. This implementation can reduce the complexity of calculations, thereby reducing the requirements for computing power for long video generation. In other words, through this implementation, long videos can be generated using computing devices with average computing power.

[0017] Based on the implementation methods provided in the above aspects, the embodiments of the present application can be further combined to provide more implementation methods.

[0018] In a second aspect, a video generation device is provided, which includes: an acquisition module, used to acquire text description information and multiple structural information of a target video, and the multiple structural information corresponds to multiple noise samples one by one; wherein each noise sample is used to generate a video frame in the target video, and the structural information corresponding to the noise sample is used to indicate the structure of the object in the video frame; and in a first time step, a first noise sample among the multiple noise samples is acquired, and the first noise sample is used to generate a first video frame in the target video; a denoising module, used to use a pre-trained diffusion model, based on the text description information and the structural information corresponding to the first noise sample, to remove the noise of the first noise sample, and obtain a first intermediate sample corresponding to the first noise sample; an updating module, used to update the features of the first intermediate sample corresponding to the first noise sample, and obtain a first output sample corresponding to the first noise sample; wherein the first output sample of the first noise sample is used as a noise sample of the next time step of the first time step, or a first video frame; wherein the updating module is used to: based on an attention mechanism, fuse the features of the first intermediate sample corresponding to the first noise sample with the features of the first intermediate sample corresponding to the other noise samples; wherein the other noise samples are used to generate other video frames in the target video except the first video frame.

[0019] In a possible implementation, the first intermediate sample corresponding to the first noise sample includes a first region and a second region, and the updating module is used to: based on an attention mechanism, fuse features of the second region with features of the first region, and fuse features of the first region with features of the second region.

[0020] In a possible implementation, the update module is used to: update the first output sample corresponding to the first noise sample based on the first output sample corresponding to the second noise sample and the first output sample corresponding to the third noise sample; wherein the second noise sample is used to generate a second video frame in the target video, the third noise sample is used to generate a third video frame in the target video, and the second video frame, the first video frame and the third video frame are adjacent in sequence in the target video.

[0021] In a possible implementation, the denoising module is used to: in the second time step, using a diffusion model, based on text description information and structural information corresponding to the first noise sample, remove the noise of the first output sample corresponding to the first noise sample, and obtain the second intermediate sample corresponding to the first noise sample; the updating module is used to: update the features of the second intermediate sample corresponding to the first noise sample, and obtain the second output sample corresponding to the first noise sample; the updating module is used to: update the second output sample corresponding to the third noise sample based on the second output sample corresponding to the first noise sample and the second output sample corresponding to the fourth noise sample among the multiple noise samples; wherein the fourth noise sample is used to generate a fourth video frame in the target video, the first video frame, the third video frame and the fourth video frame are adjacent in sequence in the target video, and the second time step is the next time step of the first time step.

[0022] In a possible implementation, a target video includes a plurality of segments connected in sequence, each segment includes a key frame and a non-key frame, and a first video frame and other video frames are key frames of different segments respectively; an updating module is used for: based on an attention mechanism, fusing a feature of a first output sample corresponding to the first noise sample and a feature of a first intermediate sample corresponding to the fifth noise sample to obtain a first output sample of the fifth noise sample; wherein the fifth noise sample and the sixth noise sample are respectively used to generate different non-key frames in the segment where the first video frame is located.

[0023] In a third aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the method provided in the first aspect.

[0024] In a fourth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method provided in the first aspect.

[0025] In a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computer device cluster, the computer device cluster executes the method provided in the first aspect.

[0026] The beneficial effects of the second to fifth aspects can be referred to the above introduction to the beneficial effects of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of a pre-trained diffusion model involved in an embodiment of the present application;

[0028] Figure 2 A schematic diagram of a video generation model provided in an embodiment of the present application;

[0029] Figure 3 A flowchart of a video generation method provided in an embodiment of the present application;

[0030] Figure 4A An edge map provided in an embodiment of the present application;

[0031] Figure 4B A depth map provided for an embodiment of the present application;

[0032] Figure 4C A target video generated for an embodiment of the present application;

[0033] Figure 5 A schematic diagram of inter-frame attention provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of a sample smoothing process provided in an embodiment of the present application;

[0035] Figure 7 A schematic diagram of a video generating device provided in an embodiment of the present application;

[0036] Figure 8 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0037] Fig. 9 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0038] Fig.10 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The scheme provided by the embodiment of the present application is described below in conjunction with the accompanying drawings. Wherein, "multiple" means two or more than two.

[0040] In the fields of film and television creation, advertising, social media, personal entertainment, etc., there are a lot of video creation or editing needs. Text-to-video technology can save time in video creation and editing, and has a wide range of application needs in these fields.

[0041] In the text-to-video generation task, the contents of each video frame that makes up the video are generated independently, without considering the correlation and continuity between the previous and next frames, resulting in poor consistency between video frames and obvious jumps.

[0042] In one solution P1, a 3D convolutional neural network (3DCNN) with an added time dimension is used to generate a video. Since the 3D convolutional neural network has many parameters, a large model, and is difficult to train, it is inconvenient to use.

[0043] In addition, in order to simplify training, in this scheme, when performing convolution in the time dimension, only the features of the current video frame, the features of the previous frame of the current video frame, and the features of the first frame of the video are convolved, that is, when generating the current video frame, a sparse spatiotemporal attention mechanism is used that only considers the previous frame of the current video frame and the first frame of the video. This leads to poor overall consistency of the video, and is prone to problems such as obvious video flickering or jumping.

[0044] In another scheme P2, a structure sequence is provided for the video generation task. When generating a video frame, the structure indicated by the structure sequence is referenced to generate the structure of the object in the video frame. In this scheme, although the consistency of the structure of the object in the video between each video frame is improved through the structure sequence, the consistency of the appearance of the object between each video frame is still poor.

[0045] An embodiment of the present application provides a video generation method, which can use a pre-trained diffusion model for generating images, combined with a structural sequence and an inter-frame attention mechanism, to generate each video frame in a target video, so that objects in the target video have good structural consistency and appearance consistency between each video frame.

[0046] The consistency of the video frames can be understood as the transition smoothness between the video frames. The higher the consistency between two adjacent video frames, the higher the transition smoothness between the two video frames, the smaller the jumpiness of the video, and the higher the quality of the video.

[0047] Next, the video generation method provided in the embodiment of the present application is introduced by way of example.

[0048] First, a diffusion model 110 is introduced. Figure 1As shown, the diffusion model 110 may include a control layer 111, at least one denoising layer 112, and at least one attention layer 113. The number of denoising layers 112 is equal to the number of attention layers 113. Moreover, the denoising layers 112 and the attention layers 113 are staggered in the data transmission direction, that is, the output of one denoising layer 112 is the input of one attention layer 113, and the output of the attention layer 113 is the input of another denoising layer 112, and so on.

[0049] The diffusion model 110 belongs to a text-image diffusion model, that is, the diffusion model 110 belongs to a model that generates images based on text description information. In some embodiments, the diffusion model 110 can be a diffusion model based on a controlnet. The text-image diffusion model can also be called a stable diffusion model.

[0050] The working process of the diffusion model 110 is as follows.

[0051] The control layer 112 may input the structural information and text description information of the target image (i.e., the image to be generated). The text description information is used to describe the content of the target image, such as the objects included in the image, the theme of the image, etc., and the structural information is used to indicate the structure of the object in the image, such as the posture and shape of the object. The control layer 111 may obtain guidance information based on the structural information and the text description information. The control layer 111 may learn the features of the structural information and the features of the text description information to obtain guidance information.

[0052] The control layer 111 may be implemented by a neural network, for example, the control layer 111 may be a residual network (resnet).

[0053] The control layer 111 can output the guidance information to each denoising layer 112. Under the action of the guidance information, the denoising layer 112 removes the noise of the noise sample. Among them, the noise sample is a sample used to generate an image. By removing the noise of the noise sample, the noise sample can be converted into an image. Specifically, when training the denoising layer 112, Gaussian noise can be gradually added to the image so that the image is called a noise sample. Then, through the denoising process, the noise in the noise sample is gradually removed, thereby restoring the image. By training the denoising layer 112, the denoising layer 112 learns the denoising process, and then the denoising layer 112 obtains the ability to generate an image by removing the noise in the noise sample.

[0054] The denoising layer 112 may be implemented by a neural network. For example, the denoising layer 112 may be a 2D convolutional neural network (2DCNN).

[0055] The denoising layer 112 removes the noise of the noise sample and obtains the intermediate sample in the image generation process. The denoising layer 112 can output the intermediate sample it generates to the attention layer 113. In the attention layer 113, based on the attention mechanism, the features of one area of ​​the intermediate sample can be integrated with the features of other areas of the intermediate sample to ensure the smoothness between different areas of the intermediate sample. The data obtained after being processed by the attention layer 113 can be referred to as an output sample. For the convenience of description, the process of integrating the features of one area of ​​the intermediate sample with the features of other areas of the intermediate sample based on the attention mechanism in the attention layer 113 is called intra-image attention.

[0056] In some embodiments, in the attention layer 113, the features of the text description information can also be integrated into the features of the intermediate samples based on the attention mechanism to ensure that the features of the intermediate samples are consistent with the content described by the text description information.

[0057] If there are multiple denoising layers 112 and attention layers 113, the attention layer 113 can output the output sample obtained by the attention layer 113 to the next denoising layer 112. The output sample is used as the noise sample of the denoising layer 112, and is denoised in the denoising layer 112 to obtain an intermediate sample. The obtained intermediate sample is input to the next attention layer 113. In the next attention layer 113, based on the attention mechanism, the features of one area of ​​the intermediate sample are integrated with the features of other areas of the intermediate sample to obtain the output sample of the attention layer 113. And so on.

[0058] The above working process can be called a time step. After multiple time steps are executed in sequence, the noise of the noise sample is removed under the guidance information obtained based on the text description information and the structural information to obtain the target image. The output sample obtained at the last time step in the multiple time step axis is the target image, and the output sample obtained at the non-last time step is the noise sample of the next time step.

[0059] The diffusion model 110 is a pre-trained model. In the video generation method provided in the embodiment of the present application, the diffusion model 110 can be used to generate a video without retraining the diffusion model 110, and the generated video has high inter-frame smoothness and low jumpiness. The details are as follows.

[0060] See also Figure 2 Based on the diffusion model 110, a video generation model 120 is designed. Figure 2As shown, the video generation model 120 includes a control layer 121, at least one denoising layer 122, and at least one attention layer 123. The number of denoising layers 122 is equal to the number of attention layers 123. Moreover, the denoising layers 122 and the attention layers 123 are staggered, that is, the output of one denoising layer 122 is the input of one attention layer 123, and the output of the attention layer 123 is the input of another denoising layer 122, and so on.

[0061] The parameter set of the control layer 121 may adopt the parameter set of the control layer 111. That is, the parameter set of the control layer 111 in the diffusion model 110 may be used as the parameter set of the control layer 121 of the video generation model 120.

[0062] The parameter set of the neural network is a set of parameters used to calculate the input data, and the parameter set includes weight parameters, bias parameters, etc. Training a neural network is essentially training the parameter set of the neural network. The parameter set of the control layer 111 in the pre-trained diffusion model 110 is used as the parameter set of the control layer 121 of the video generation model 120, so that the control layer 121 can be obtained without training.

[0063] Similar to the text description information of the target image, the text description information of the target video is used to describe the content of the target video.

[0064] The multiple structural information and the multiple noise samples correspond one to one, each noise sample is used to generate a video frame in the target video, and the multiple noise samples can generate multiple video frames, and the multiple video frames constitute the target video.

[0065] Similar to the structural information of the target image, the structural information is used to indicate the structure of an object in a video frame of the target video, wherein the video frame is a video frame to be generated by the noise sample corresponding to the structural information.

[0066] The control layer 121 can obtain the guidance information corresponding to the noise sample based on the text description information and the structural information corresponding to the noise sample. And output the guidance information corresponding to the noise sample to each denoising layer 122. Under the action of the guidance information corresponding to the noise sample, the denoising layer 122 removes the noise of the noise sample and obtains the intermediate sample corresponding to the noise sample. The denoising layer 122 performs the above-mentioned processing on each noise sample to obtain multiple intermediate samples corresponding to the multiple noise samples, and the multiple noise samples correspond to the multiple intermediate samples one-to-one.

[0067] The parameter set of the denoising layer 122 may adopt the parameter set of the denoising layer 112. That is, the parameter set of the denoising layer 112 in the diffusion model 110 may be used as the parameter set of the denoising layer 122 of the video generation model 120. This achieves obtaining the denoising layer 122 without training.

[0068] In some embodiments, the denoising layer 112 may be a 2D convolutional network, and the denoising layer 122 may be obtained by expanding the U-Net of the denoising layer 121 along the time axis. Specifically, the size of the convolution kernel of the 2D convolutional network as the denoising layer 112 may be set to M×M, and the convolution kernel may be replaced with a 1×M×M convolution kernel to complete the expansion of the U-Net on the time axis. The M×M convolution kernel is replaced with a 1×M×M convolution kernel, which can meet the requirements of the timing of the video frame on the network structure, and there is no need to adjust the parameter set for the 2D convolution operation, so that the denoising layer 122 is obtained without retraining.

[0069] The denoising layer 122 can output the intermediate samples corresponding to the obtained noise samples to the attention layer 123. In the attention layer 123, based on the attention mechanism, the features of the intermediate samples corresponding to the noise samples can be fused with the features of the intermediate samples corresponding to other noise samples to update the features of the intermediate samples corresponding to the noise samples and obtain the output samples of the noise samples. That is to say, in the attention layer 123, the features of the intermediate samples corresponding to the noise samples are updated while considering the features of the intermediate samples corresponding to other noise samples to obtain the output samples of the noise samples. For the convenience of description, the process of fusing the features of the intermediate samples corresponding to other noise samples with the features of the intermediate samples corresponding to the noise samples based on the attention mechanism is called inter-frame attention. Through inter-frame attention, the consistency between different video frames of the target video can be improved and the jumping feeling of the target video can be reduced.

[0070] Attention layer 123 is obtained based on attention layer 113. Specifically, the parameter set of attention layer 113 is used as the parameter set of attention layer 123. That is, in attention layer 123, inter-frame attention is implemented using the parameter set used in attention layer 113 to implement intra-image attention. Thus, attention layer 123 is obtained without training.

[0071] In some embodiments, in the attention layer 123, based on the attention mechanism, the features of one region of the intermediate sample can be fused with the features of other regions of the intermediate sample to improve the consistency between different regions of the intermediate sample. For the convenience of description, the process of fusing the features of one region of the intermediate sample with the features of other regions of the intermediate sample based on the attention mechanism in the attention layer 123 is called intra-frame attention.

[0072] In some embodiments, in the attention layer 123, the features of the text description information can also be integrated into the features of the intermediate samples based on the attention mechanism to ensure that the features of the intermediate samples are consistent with the content described by the text description information.

[0073] The result obtained by performing data processing on the intermediate sample of the noise sample in attention 123 can be called the output sample corresponding to the noise sample.

[0074] If there are multiple denoising layers 122 and attention layers 123, the attention layer 123 can output the output sample obtained by the attention layer 122 to the next denoising layer 122. The output sample is used as the noise sample of the denoising layer 122, and is denoised in the denoising layer 122 to obtain an intermediate sample. The obtained intermediate sample is input to the next attention layer 123. In the next attention layer 123, the features of the intermediate sample can be updated according to the inter-frame attention and the inter-frame attention to obtain the output sample. And so on.

[0075] The above working process can be called a time step. After multiple time steps are executed in sequence, the noise of the noise sample is removed under the action of the text description information and the structural information to obtain the video frame of the target video. Among them, the output sample obtained in the last time step of the multiple time step axes is the video frame, and the output sample obtained in the non-last time step is the noise sample of the next time step.

[0076] In some embodiments, Figure 2 As shown, the video generation model 122 may further include a smoothing layer 124. In the smoothing layer 124, the features of the output samples of the noise samples on both sides of the current noise sample may be used to update the features of the output samples of the current noise sample, so as to improve the consistency between adjacent video frames in the generated target video frame. The noise samples on both sides of the current noise sample refer to noise samples used to generate video frames located on both sides of the current video frame, wherein the current video frame refers to a video frame generated by the current noise sample. The function of the smoothing layer 124 will be specifically introduced in the method embodiment below, and will not be repeated here.

[0077] The smoothing layer 124 does not need to be implemented by a neural network, but can be implemented by a feature calculation tool of an image or video frame, that is, the smoothing layer 124 does not need to be obtained by training.

[0078] In summary, based on the pre-trained diffusion model 110, text-to-video generation can be achieved without model training, and the consistency between different frames of the generated video can be guaranteed, thereby reducing the jumpiness of the video.

[0079] The above examples introduce a model for video generation. Next, the video generation method provided by the embodiment of the present application is introduced.

[0080] The video generation method provided in the embodiments of the present application can be executed by any device, equipment, platform, or device cluster with data processing capabilities.

[0081] See also Figure 3 The video generation method provided in the embodiment of the present application may include the following steps.

[0082] Step 301, obtaining text description information and multiple structural information of a target video.

[0083] The text description information of the target video is used to describe the content of the target video, such as the objects included in the target video and the appearance of the objects. Another example is the style and theme of the target video. The appearance of the object includes the color, brightness and local structural features of the object.

[0084] Multiple structural information corresponds to multiple noise samples, and one structural information corresponds to one of the multiple noise samples, and different structural information corresponds to different noise samples. In short, multiple structural information and multiple noise samples correspond one to one. The noise sample corresponding to the structural information is used to generate a video frame of the target video, and the structural information is used to indicate the structure of the object in the video frame. The structure of the object in the video frame may refer to geometric features such as the posture and shape of the object in the video frame.

[0085] In some embodiments, the structural information may specifically be edge features, depth features, or skeleton features of human body postures. In some embodiments, an edge map may be used as structural information, for example Figure 4A The edge map shown in FIG. An edge map is used as structural information and corresponds to a noise sample. The edge map can be extracted from an existing image or video frame, or drawn. In some embodiments, a depth map can be used as structural information, for example Figure 4B The depth map shown. Wherein, a depth map serves as structural information and corresponds to a noise sample. Wherein, the depth map can be extracted from an existing image or video frame, or drawn. In some embodiments, a posture guidance map can be used as structural information. Wherein, a posture guidance map serves as structural information and corresponds to a noise sample. Wherein, the posture guidance map can be extracted from an existing image or video frame, or drawn.

[0086] Continue reading Figure 3 In step 302, multiple noise samples corresponding to multiple structural information are obtained in time step t. For example, multiple noise samples such as noise sample A1 are obtained.

[0087] Wherein, time step t is any one of the above-mentioned multiple time steps. As described above, the generated target video may be composed of multiple time steps executed in sequence. If time step t is the first time step executed among the multiple time steps, the noise sample obtained in step 302 is a randomly generated pure noise sample. If time step t is not the first time step executed among the multiple time steps, the noise sample is the output sample obtained in the previous time step. That is, the output sample obtained in a time step is the noise sample of the next time step of the time step.

[0088] Next, in step 303, the pre-trained diffusion model is used to remove the noise of the noise sample based on the text description information and the structural information corresponding to the noise sample to obtain an intermediate sample corresponding to the noise sample; and the features of the intermediate sample corresponding to the noise sample are updated to obtain an output sample corresponding to the noise sample.

[0089] Among them, step 303 can be performed by the control layer 121, the denoising layer 122, and the attention layer 123 in the video generation model 120. As described above, the control layer 121, the denoising layer 122, and the attention layer 123 in the video generation model 120 are respectively obtained based on the control layer 111 and the denoising layer 112 in the pre-trained diffusion model 110. Therefore, the operation of step 303 can be called using the pre-trained diffusion model 110, based on the text description information and the structural information corresponding to the noise sample, to remove the noise of the noise sample to obtain the intermediate sample corresponding to the noise sample; and updating the features of the intermediate sample corresponding to the noise sample to obtain the output sample corresponding to the noise sample.

[0090] The specific execution process of step 303 is as follows.

[0091] Specifically, the text description information and multiple structural information obtained in step 301 are input to the control layer 121 of the video generation model 120. The control layer 121 can extract the features of the text description information and the features of each structural information to obtain the guidance information of the noise sample corresponding to the structural information. For example, taking the noise sample A1 as an example, the control layer 121 can obtain the guidance information of the noise sample A1 based on the features of the text description information and the features of the structural information corresponding to the noise sample A1.

[0092] The denoising layer 122 can obtain multiple noise samples, and remove the noise of the noise samples under the guidance information of each noise sample to obtain an intermediate sample of the noise sample. For example, the denoising layer 122 removes the noise of the noise sample A1 under the guidance information of the noise sample A1. Here, removing the noise of the noise sample can be understood as adjusting the noise distribution or pixel distribution in the noise sample under the guidance information, so that the noise or pixel distribution in the noise sample approaches the pixel distribution of the video frame to generate a video frame.

[0093] In some embodiments, text description information may be directly input to the denoising layer 122. The text description information input to the denoising layer 122 may be a copy of the text description information input to the control layer 121. Under the guidance information, the denoising layer 122 considers the text description information when removing noise from the noise sample, so that the noise sample after noise removal is closer to the content described by the text description information.

[0094] The noise of the noise sample is removed in the denoising layer 122 to obtain an intermediate sample corresponding to the noise sample. For example, after the noise of the noise sample A1 is removed, an intermediate sample corresponding to the noise sample A1 can be obtained.

[0095] Among them, if time step t is the last time step among multiple time steps, the output sample obtained at time step t is the video frame. If time step t is not the last time step among multiple time steps, the output sample obtained at time step t is the noise sample in the next time step of time step t. The next time step of time step t refers to the time step executed after time step t.

[0096] Step 303 may be performed by the attention layer 123. Specifically, if the output sample is the output sample of the last attention layer 123 among the multiple attention layers 123, the output sample is used as the output sample of the time step t. If the output sample is the output sample of the non-last attention layer 123 among the multiple attention layers 123, the output sample is used as the noise sample of the denoising layer 122 after the attention layer 123.

[0097] Updating the features of the intermediate samples corresponding to the noise samples may include updating the features of the intermediate samples based on inter-frame attention. Specifically, taking the noise sample A1 as an example, based on the attention mechanism, the features of the intermediate samples corresponding to the noise sample A1 are fused with the features of the intermediate samples corresponding to other noise samples. The other noise samples are the noise samples other than the noise sample A1 among the above-mentioned multiple noise samples. That is, the other noise samples are used to generate other video frames in the target video except the video frame generated by the noise sample A1.

[0098] In the following, for the convenience of description, when the features of the intermediate samples corresponding to the noise sample A1 are updated based on the attention mechanism, the noise samples corresponding to the intermediate samples considered are referred to as the focused noise samples of the noise sample A1. That is, in step 303, the features of the intermediate samples corresponding to the focused noise samples of the noise sample A1 are fused with the features of the intermediate samples corresponding to the noise sample A1 based on the attention mechanism to obtain the output sample of the noise sample A1.

[0099] In some embodiments, the noise sample of interest of the noise sample A1 may be all noise samples except the noise sample A1 among the multiple noise samples, that is, the features of all noise samples except the noise sample A1 are merged into the features of the noise sample A1. This improves the consistency between the video frame generated by the noise sample A1 and the video frame generated by the noise sample of interest of the noise sample A1.

[0100] In particular, when the target video is a short video, that is, when the computing power requirement is not high, the features of all noise samples other than noise sample A1 can be fused into the features of noise sample A1 to improve the consistency between the video frames generated by noise sample A1 and the video frames generated by the focused noise samples of noise sample A1.

[0101] In an example of this embodiment, updating the features of the intermediate sample corresponding to the noise sample may include updating the features of the intermediate sample based on intra-frame attention. Specifically, based on the attention mechanism, the features of one region of the intermediate sample corresponding to the noise sample may be fused with features of other regions of the intermediate sample. For example, the intermediate sample has region B1 and region B2, and the features of region B2 may be fused with the features of region B1, and the features of region B1 may be fused with the features of region B2. In this way, the consistency between different regions of the intermediate sample may be improved.

[0102] like Figure 5 As shown, this embodiment uses inter-frame attention and intra-frame attention to update the features of the intermediate sample to obtain the output sample. Compared with the output sample using only intra-frame attention, the video frame generated by the output sample using inter-frame attention and intra-frame attention has high consistency with other video frames in the target video, and the consistency between different regions in the video frame generated by the output sample is also high.

[0103] in, Figure 5 middle, They respectively represent an intermediate sample corresponding to a noise sample among multiple noise samples, and N is the total number of noise samples.

[0104] In one example, the intermediate samples corresponding to the above-mentioned multiple noise samples can be updated based on the inter-frame attention and intra-frame attention features by using equations (1) and (2) to obtain the output samples corresponding to the above-mentioned multiple noise samples.

[0105]

[0106] Among them, Q stands for query, K stands for key, V stands for value, and W q represents the weight of the query, W k Represents the weight of the key, W vrepresents the weight of the value. d represents the vector dimension of the feature vector of the intermediate sample, the superscript T represents the transpose, the subscript t represents the tth time step, N represents the number of intermediate samples (i.e., the number of noise samples or video frames in the target video), z t represents the middle sample, represents the i-th intermediate sample, z′ t Represents the output sample.

[0107] In some embodiments, see Figure 2 , the attention layer 123 can output the output samples calculated by it to the smoothing layer 124, and the output samples can be smoothed in the smoothing layer 124, as follows.

[0108] The features of the output samples of the noise samples on both sides of the current noise sample can be used to update the features of the output samples of the current noise sample. The current noise sample refers to the processing object of the current smoothing process. The noise samples on both sides of the current noise sample refer to the noise samples used to generate the video frames located on both sides of the video frame generated by the current noise sample.

[0109] Taking the current noise sample as noise sample A1 as an example, the video frame generated by noise sample A2 and the video frame generated by noise sample A3 can be set to be located on both sides of the video frame generated by noise sample A1 in the target video. That is, in the target video, the video frame generated by noise sample A2, the video frame generated by noise sample A1, and the video frame generated by noise sample A3 are adjacent in sequence. The output sample corresponding to noise sample A1 can be updated by using the output sample corresponding to noise sample A2 and the output sample corresponding to noise sample A3.

[0110] Among them, when the video frame generated by the noise sample A1 is an odd frame in the target video, the video frame generated by the noise sample A2 and the video frame generated by the noise sample A3 are even frames in the target video. When the video frame generated by the noise sample A1 is an even frame in the target video, the video frame generated by the noise sample A2 and the video frame generated by the noise sample A3 are odd frames in the target video. In other words, the output samples of the noise samples corresponding to the even frames in the target video can be used to update the output samples of the noise samples corresponding to the odd frames in the target video; or, the output samples of the noise samples corresponding to the even frames in the target video can be used to update the output samples of the noise samples corresponding to the odd frames in the target video. This can be called odd-even interlaced smoothing.

[0111] In addition, when there are multiple attention layers 123, the output sample for smoothing (the output sample corresponding to the noise sample A1) is the output sample output by the last attention layer 123 among the multiple attention layers 123. The output sample output by the last attention layer 123 can also be called a video frame latent variable.

[0112] In some embodiments, as described above, from the original noise sample (i.e., the pure noise sample after denoising) to the video frame, multiple time steps executed in sequence are required. The output samples can be smoothed in two consecutive time steps respectively. The two consecutive time steps can be set to be time step t and time step t-1. Among them, time step t-1 is executed after time step t, that is, time step t-1 is the next time step of time step t. If in time step t, the output samples of the noise samples corresponding to the odd frames in the target video are used to update the output samples of the noise samples corresponding to the even frames in the target video, then in time step t-1, the output samples of the noise samples corresponding to the even frames in the target video can be used to update the output samples of the noise samples corresponding to the odd frames in the target video. In this way, it is ensured that in two consecutive time steps, the output samples corresponding to each non-edge noise sample are smoothed. Among them, the non-edge noise sample refers to the video frame generated by the noise sample is not the first frame of the target video, nor the last frame of the target video.

[0113] Among them, the specific process of smoothing the output samples in two consecutive time steps is as follows.

[0114] For the convenience of description, the output sample obtained at time step t can be called the first output sample, and the output sample obtained at time step t-1 can be called the second output sample.

[0115] After the end of time step t, that is, after smoothing the first output sample corresponding to the noise sample and updating the first output sample corresponding to the noise sample, time step t-1 is started. The first output sample corresponding to the noise sample obtained at time step t1 is the noise sample at time step t-1. At time step t-1, based on the pre-trained diffusion model, based on the above text description information and the structural information corresponding to the noise sample, the noise of the first output sample of the noise sample can be removed to obtain the intermediate sample corresponding to the noise sample in time step t-1. The specific process can be implemented with reference to the above description, and will not be repeated here.

[0116] The features of the intermediate samples corresponding to the noise samples in time step t-1 can be updated to obtain the second output samples corresponding to the noise samples. The specific process can be implemented by referring to the introduction of step 303 above, which will not be described here.

[0117] After obtaining the second output samples corresponding to the above-mentioned multiple noise samples, smoothing processing can be performed. For example, it can be set that in the target video, the video frame generated by the noise sample A2, the video frame generated by the noise sample A1, the video frame generated by the noise sample A3, and the video frame generated by the noise sample A3 are adjacent in sequence. Among them, in time step t, the first output sample corresponding to the noise sample A1 is updated using the first output sample corresponding to the noise sample A2 and the first output sample corresponding to the noise sample A3. Then in time step t-1, the first output sample corresponding to the noise sample A3 is updated using the first output sample corresponding to the noise sample A1 and the first output sample corresponding to the noise sample A4. In this way, interlaced smoothing of odd and even frames is achieved, which improves the consistency between adjacent video frames while avoiding the content of adjacent video frames being too similar.

[0118] For an illustrative example, see Figure 6 , the output sample corresponding to the current noise sample can be updated by interpolation. Specifically, the feature vectors of the output samples of the noise samples on both sides of the current noise sample are used to calculate (for example, sum or average) a fused feature vector, and the output sample represented by the fused feature vector is used as the output sample corresponding to the updated current noise sample.

[0119] More specifically, if Figure 6 As shown, the above-mentioned multiple noise samples can be set to include noise sample i-3, noise sample i-2, noise sample i-1, noise sample i, noise sample i+1, noise sample i+2, noise sample i+3, etc. The video frame generated by the noise sample i-3, the video frame generated by the noise sample i-2, the video frame generated by the noise sample i-1, the video frame generated by the noise sample i, the video frame generated by the noise sample i+1, the video frame generated by the noise sample i+2, and the video frame generated by the noise sample i+3 are sequentially adjacent in the target video.

[0120] At time step t, the output sample corresponding to noise sample i-3 and the output sample corresponding to noise sample i-1 are used to update the output sample corresponding to noise sample i-2, the output sample corresponding to noise sample i-1 and the output sample corresponding to noise sample i+1 are used to update the output sample corresponding to noise sample i, and the output sample corresponding to noise sample i+2 is updated using the output sample corresponding to noise sample i+1 and the output sample corresponding to noise sample i+3.

[0121] Then, at time step t-1, the output samples corresponding to noise sample i-2 and the output samples corresponding to noise sample i are used to update the output samples corresponding to noise sample i-1, and the output samples corresponding to noise sample i and the output samples corresponding to noise sample i+2 are used to update the output samples corresponding to noise sample i+1.

[0122] Thus, the output sample corresponding to each non-edge noise sample can be smoothed.

[0123] In one example, the output sample can be projected into an image, that is, the feature vector of the output sample is converted into a feature vector in an image format (e.g., RGB format). In this way, the features of the output sample can be directly updated by using the interpolation operation in the image field. At time step t, the feature vector of the output sample can be converted into a feature vector in an image format by using formula (3).

[0124] x t→0 =D(z t→0 ) (3)

[0125] Among them, z t→0 represents the feature vector of the output sample output by the attention layer 123 at time step t, x t→0 represents the feature vector of the image projected by the output sample output by the attention layer 123 at time step t, and D represents a decoder, which is used to convert the data in the data format of the attention layer 123 into data in the image format. The data format of the attention layer 123 refers to the data format of the data output by the attention layer 123, which is generally a latent space coding format.

[0126] Then, using x t→0 Perform interpolation operation to obtain Taking the output sample corresponding to noise sample i-1 and the output sample corresponding to noise sample i+1 as an example, updating the output sample corresponding to noise sample i, using the output sample corresponding to noise sample i-1 t→0 and x corresponding to noise sample i-1 t→0 , and obtain the corresponding noise sample i Then Convert to the data format of attention layer 123 to obtain the feature vector of the output sample after the noise sample i is updated

[0127] Among them, the feature vector in the image format can be converted into a feature vector in the data format of the attention layer 123 through formula (4).

[0128]

[0129] Among them, E represents an encoder, which is used to convert data in image format into data in the data format of the attention layer 123.

[0130] Similarly, at time step t-1, the feature vector z of the output sample is t-1→0 Convert the feature vector x to image format t-1→0 Then, using x t-1→0 Perform interpolation operation to obtain Taking the output sample corresponding to noise sample i and the output sample corresponding to noise sample i+2 as an example, updating the output sample corresponding to noise sample i+1, using the output sample corresponding to noise sample i t-1→0 and x corresponding to noise sample i+2 t-1→0 , and obtain the corresponding noise sample i+1 Then Convert to the data format of attention layer 123 to obtain the feature vector of the updated output sample of noise sample i+1

[0131] In some embodiments, the time step t and the time step t-1 for performing the smoothing operation are intermediate steps in the multiple time steps. In one example, the total number of time steps in the multiple time steps can be set to T. When T is an odd number, t is equal to one-half of (T+1). When T is an even number, t or t-1 is equal to one-half of T.

[0132] As described above, in step 303, based on the attention mechanism, the features of all noise samples other than noise sample A1 can be fused into the features of noise sample A1. When the target video is a long video, this feature fusion method requires high computing power. In order to reduce the computing power requirements for long video generation, step 303 can also be implemented by the following segment fusion feature solution.

[0133] The target video can be divided into multiple video segments. The video segments can be referred to as segments. At least one key frame can be selected for each segment, and the frames in the segment other than the key frames are called non-key frames. In other words, the noise sample sequence corresponding to the target video is divided into multiple sample segments. The noise sample sequence is obtained by sorting multiple noise samples according to the positions of the video frames to be generated by the noise samples in the target video. At least one key noise sample can be selected for each sample segment. In this way, multiple key noise samples can be obtained for multiple sample segments. The key noise sample is a noise sample used to generate a key frame, and accordingly, the noise sample used to generate a non-key frame can be called a non-key noise sample.

[0134] In an example, the key noise sample of a sample segment is the first noise sample or the last noise sample in the sample segment.

[0135] When updating the features of the intermediate samples corresponding to the noise samples (i.e., the key noise samples) used to generate the key frames, only the features of other key noise samples are considered. That is, taking the updating of the features of the intermediate samples corresponding to the noise sample A1 in step 303 as an example, if the noise sample A1 is the key noise sample, then when updating the features of the intermediate samples of the noise sample A1 in step 303, the noise samples of interest of the noise sample A1 are the key noise samples other than the noise sample A1 among the above-mentioned multiple key noise samples. That is, the video frame generated by the noise sample A1 and the video frame generated by the noise sample A1's focus noise sample belong to the same key frame.

[0136] In this way, the features of the intermediate samples corresponding to the key noise samples of each sample segment are integrated with the features of the intermediate samples corresponding to the key noise samples of other sample segments to obtain the output samples for generating key frames, which can ensure the consistency between the video segments corresponding to different sample segments.

[0137] In step 303, when updating the features of the intermediate samples corresponding to the noise samples (i.e., non-critical noise samples) used to generate non-key frames, other noise samples in the sample segment where the noise sample is located are considered. That is, taking the updating of the features of the intermediate samples corresponding to the noise sample A1 in step 303 as an example, if the noise sample A1 is a non-critical noise sample, the noise samples of interest for the noise sample A1 include other noise samples other than the noise sample A1 in the sample segment where the noise sample A1 is located, that is, all or part of the video frames generated by the noise sample A1 and the noise samples of interest for the noise sample A1 belong to the same video segment. Furthermore, the noise samples of interest for the noise sample A1 include the critical noise samples of the sample segment where the noise sample A1 is located.

[0138] In one example, the noise sample A1 is a non-critical noise sample, and the noise samples of interest for the noise sample A1 are: all noise samples except the noise sample A1 in the sample segment where the noise sample A1 is located. It can be understood that all noise samples except the noise sample A1 in the sample segment where the noise sample A1 is located include the critical noise samples of the sample segment where the noise sample A1 is located.

[0139] In one example, the key noise sample of a sample segment is the first noise sample in the sample segment. The noise samples of interest of noise sample A1 are: all noise samples except noise sample A1 in the sample segment where noise sample A1 is located and the key frame of the next sample segment. The next sample segment refers to the sample segment after the sample segment where noise sample A1 is located in the above noise sample sequence.

[0140] In one example, the key noise sample of a sample segment is the last noise sample in the sample segment. The noise samples of interest of noise sample A1 are: all noise samples except noise sample A1 in the sample segment where noise sample A1 is located and the key frame of the previous sample segment. The previous sample segment refers to the sample segment before the sample segment where noise sample A1 is located in the above noise sample sequence.

[0141] Among them, the noise sample A1 is a non-critical noise sample. When the intermediate samples corresponding to the noise sample A1 are updated based on the attention mechanism to obtain the output sample corresponding to the noise sample A1, the features of the intermediate samples corresponding to the non-critical noise samples in the focused noise samples of the noise sample A1 and the features of the output samples of the critical noise samples are fused into the features of the intermediate samples corresponding to the noise sample A1 to obtain the output sample corresponding to the noise sample A1.

[0142] Therefore, in the features of the output samples corresponding to the non-critical noise samples of each sample segment, the features of the intermediate samples corresponding to other noise samples of the sample segment are integrated, so that the consistency between the video frames in the video segment corresponding to the sample segment can be ensured. The features of the output samples of the critical noise samples have been integrated with the features of other critical noise samples. In the features of the output samples corresponding to the non-critical noise samples of each sample segment, the features of the output samples of the critical noise samples are integrated, so that the consistency between different video segments can be ensured.

[0143] In one example, the segmented fusion feature solution introduced above can be implemented by the following formula.

[0144] Specifically, the multiple noise samples shown in formula (5) can be divided into Figure 6 Several sample clips are shown.

[0145]

[0146]

[0147] Among them, z t represents the intermediate samples corresponding to multiple noise samples, N represents the total number of noise samples in multiple noise samples, t represents the tth time step, Represents the middle sample corresponding to the i-th noise sample among multiple noise samples, represents the pth sample segment among multiple sample segments, N c represents the length of the sample segment (i.e., the number of noise samples in the sample segment), Represents the middle sample corresponding to a noise sample in sample segment p.

[0148] The key noise sample can be selected for the sample segment by using formula (7).

[0149]

[0150] in, The intermediate sample corresponding to the key noise sample of the sample segment p represents the key noise sample of each sample segment, wherein the key noise sample of each sample segment is the first noise sample of the sample segment.

[0151] The output samples corresponding to the key noise samples can be calculated by using formula (8) and formula (9).

[0152]

[0153]

[0154] Among them, Q key represents the query of the intermediate samples corresponding to the key noise samples, K key represents the key of the intermediate sample corresponding to the key noise sample, V key Represents the value of the intermediate sample corresponding to the key noise sample, W q represents the weight of the query, W k Represents the weight of the key, W v represents the weight of the value. d represents the vector dimension of the feature vector of the intermediate sample, the superscript T represents the transpose, and z t ′key Represents the output sample corresponding to the key noise sample.

[0155] The output samples corresponding to the non-critical noise samples can be calculated by using formula (10) and formula (11).

[0156]

[0157]

[0158] Among them, Q p represents the query of the intermediate sample corresponding to the p-th sample segment, K p represents the key of the intermediate sample corresponding to the p-th sample fragment, V p represents the value of the middle sample corresponding to the p-th sample segment, W q represents the weight of the query, W k Represents the weight of the key, W v The weight of the representative value, represents the interval consisting of the key noise samples in the p-th sample segment and the key noise samples in the p+1-th sample segment, z′ t prepresents the output sample corresponding to the pth sample segment. When calculating the output sample corresponding to the non-critical noise sample, the output sample of the key frame is fused into the intermediate sample of the non-critical noise sample.

[0159] The intermediate samples corresponding to the p-th segment include the intermediate samples corresponding to each noise sample in the p-th segment. The output samples corresponding to the p-th segment include the output samples corresponding to each noise sample in the p-th segment.

[0160] In summary, the video generation method provided in the embodiment of the present application does not require training of a video generation model, and can generate videos using a pre-trained diffusion model, and the video frames of the generated videos have good consistency.

[0161] Among them, "a swan moves in the lake" is used as the text description information, Figure 4A The edge diagram shown or Figure 4B The depth map shown is structural information. According to the video generation method provided in the embodiment of the present application, Figure 4C Target video shown.

[0162] Through simulation experiments, it is found that compared with the schemes P1 and P2 described above, the video generated by the video generation method provided in the embodiment of the present application has better frame consistency.

[0163] We obtained 25 object-centric videos. We annotated the description of each video using a bootstrapped model (BLIP-2) with frozen image encoders and large language models for language-image pre-training. Based on the description of each video, we used

[0164] AI generated content (AIGC) technology generates 5 text descriptions for each video, thus obtaining 125 video-text pairs consisting of text description and target video.

[0165] The video generation method, scheme P1 and scheme P2 provided in the embodiments of the present application are respectively adopted to generate the target video using 125 video-text pairs.

[0166] The video generation method and solution P2 provided in the embodiment of the present application both use edge maps and depth maps as structural information to generate the target video. Solution P1 uses denoising diffusion implicit models (DDIM) inversion to generate the target video.

[0167] Then, the contrast language-image pre-training (CLIP) model is used to evaluate the quality of the target video from two aspects: frame consistency and promote consistency. Frame consistency refers to the average similarity between all adjacent frame pairs, and promote consistency refers to the average similarity between the content described by the text description information and the content of all video frames. The evaluation structure is shown in Table 1.

[0168] Table 1

[0169]

[0170] As shown in Table 1, when the edge map is used as structural information, the frame consistency of the target video generated by the video generation method provided by the embodiment of the present application is 5.3 percentage points higher than the frame consistency of the target video generated by scheme P2, and 2.3 percentage points higher than the frame consistency of the target video generated by scheme P1. When the depth map is used as structural information, the frame consistency of the target video generated by the video generation method provided by the embodiment of the present application is 1.3 percentage points higher than the frame consistency of the target video generated by scheme P2, and 2.7 percentage points higher than the frame consistency of the target video generated by scheme P1.

[0171] Therefore, it can be known that the target video generated by the video generation method provided in the present application has good frame consistency.

[0172] The embodiment of the present application also provides a video generating device 700. Figure 7 As shown, the apparatus 700 includes:

[0173] The acquisition module 710 is used to acquire text description information and multiple structural information of the target video, wherein the multiple structural information corresponds to multiple noise samples one by one; wherein each of the noise samples is used to generate a video frame in the target video, and the structural information corresponding to the noise sample is used to indicate the structure of the object in the video frame; and in a first time step, acquire a first noise sample from the multiple noise samples, wherein the first noise sample is used to generate a first video frame in the target video;

[0174] A denoising module 720, configured to use a pre-trained diffusion model to remove noise from the first noise sample based on the text description information and the structural information corresponding to the first noise sample, so as to obtain a first intermediate sample corresponding to the first noise sample;

[0175] An updating module 730 is configured to update a feature of a first intermediate sample corresponding to the first noise sample to obtain a first output sample corresponding to the first noise sample; wherein the first output sample of the first noise sample is used as a noise sample of a next time step of the first time step, or the first video frame;

[0176] The update module 730 is used to: based on the attention mechanism, fuse the features of the first intermediate sample corresponding to the first noise sample with the features of the first intermediate sample corresponding to the first noise sample; wherein the other noise samples are used to generate other video frames in the target video except the first video frame.

[0177] Among them, the acquisition module 710, the denoising module 720 and the updating module 730 can all be implemented by software, or can be implemented by hardware. Exemplarily, the following takes the acquisition module 710 as an example to introduce the implementation of the acquisition module 710. Similarly, the implementation of the denoising module 720 and the updating module 730 can refer to the implementation of the acquisition module 710.

[0178] As an example of a software functional unit, the acquisition module 710 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the acquisition module 710 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0179] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0180] As an example of a hardware functional unit, the acquisition module 710 may include at least one computing device, such as a server, etc. Alternatively, the acquisition module 710 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0181] The multiple computing devices included in the acquisition module 710 can be distributed in the same region or in different regions. The multiple computing devices included in the acquisition module 710 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the acquisition module 710 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0182] It should be noted that, in other embodiments, the acquisition module 710 may be used to execute Figure 3 In any step of the method shown, the denoising module 720 can be used to perform Figure 3 In any step of the method shown, the update module 730 can be used to perform Figure 3 The steps implemented by the acquisition module 710, the denoising module 720 and the updating module 730 can be specified as needed, and the acquisition module 710, the denoising module 720 and the updating module 730 can be respectively implemented Figure 3 The different steps in the method shown are used to implement the overall functions of the apparatus 700 .

[0183] The present application also provides a computing device 800. Figure 8 As shown, the computing device 800 includes: a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate through the bus 802. The computing device 800 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 800.

[0184] The bus 802 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus 802 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 802 may include a path for transmitting information between various components of the computing device 800 (eg, the memory 806, the processor 804, and the communication interface 808).

[0185] The processor 804 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0186] The memory 806 may include a volatile memory, such as a random access memory (RAM). The memory 806 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0187] The memory 806 stores executable program codes, and the processor 804 executes the executable program codes to respectively implement the functions of the acquisition module 710, the denoising module 720, and the updating module 730, thereby implementing Figure 3 That is, the memory 806 stores the method for executing Figure 3 Instructions for the method shown.

[0188] The communication interface 808 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 800 and other devices or communication networks.

[0189] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0190] like Fig. 9 As shown, the computing device cluster includes at least one computing device 800. The memory 806 in one or more computing devices 800 in the computing device cluster may store the same Figure 3 Instructions for the method shown.

[0191] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store a program for executing Figure 3 In other words, the combination of one or more computing devices 800 can jointly execute instructions for executing Figure 3 Instructions for the method shown.

[0192] It should be noted that the memory 806 in different computing devices 800 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the apparatus 700. That is, the instructions stored in the memory 806 in different computing devices 800 may implement the functions of one or more modules in the acquisition module 710, the denoising module 720 and the updating module 730.

[0193] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.10 A possible implementation is shown. Fig.10 As shown, two computing devices 800A and 800B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 806 in the computing device 800A stores instructions for executing the functions of the acquisition module 710. At the same time, the memory 806 in the computing device 800B stores instructions for executing the functions of the denoising module 720 and the updating module 730.

[0194] It should be understood that Fig.10 The functions of the computing device 800A shown in FIG. 8 may also be completed by multiple computing devices 800. Similarly, the functions of the computing device 800B may also be completed by multiple computing devices 800.

[0195] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig. 9 and Fig.10 The difference is that the memory 806 in one or more computing devices 800 in the computing device cluster may store the same memory for executing Figure 3 Instructions for the method shown.

[0196] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store a program for executing Figure 3 In other words, the combination of one or more computing devices 800 can jointly execute instructions for executing Figure 3 Instructions for the method shown.

[0197] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes Figure 3 The method shown.

[0198] The present application also provides a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by a computing device or a host migration device such as a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute Figure 3 The method shown.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video generation method, characterized in that: The method comprises: Acquire text description information and multiple structural information of a target video, wherein the multiple structural information corresponds to multiple noise samples one by one; wherein each of the noise samples is used to generate a video frame in the target video, and the structural information corresponding to the noise sample is used to indicate the structure of an object in the video frame; In a first time step, obtaining a first noise sample from the plurality of noise samples, wherein the first noise sample is used to generate a first video frame in the target video; Using a pre-trained diffusion model, based on the text description information and the structural information corresponding to the first noise sample, removing the noise of the first noise sample to obtain a first intermediate sample corresponding to the first noise sample; and updating the features of the first intermediate sample corresponding to the first noise sample to obtain a first output sample corresponding to the first noise sample; wherein the first output sample of the first noise sample is used as a noise sample of a next time step of the first time step, or the first video frame; Among them, updating the features of the first intermediate sample corresponding to the first noise sample includes: based on the attention mechanism, fusing the features of the first intermediate sample corresponding to the first noise sample with the features of the first intermediate sample corresponding to other noise samples; wherein the other noise samples are used to generate other video frames in the target video except the first video frame.

2. The method according to claim 1, characterized in that The first intermediate sample corresponding to the first noise sample includes a first area and a second area, and the feature of updating the first intermediate sample corresponding to the first noise sample further includes: Based on the attention mechanism, features of the second region are fused with features of the first region, and features of the first region are fused with features of the second region.

3. The method according to claim 1 or 2, characterized in that: The method further includes: updating the first output sample corresponding to the first noise sample based on the first output sample corresponding to the second noise sample and the first output sample corresponding to the third noise sample; The second noise sample is used to generate a second video frame in the target video, the third noise sample is used to generate a third video frame in the target video, and the second video frame, the first video frame and the third video frame are sequentially adjacent in the target video.

4. The method according to claim 3, characterized in that The method further comprises: In a second time step, using the diffusion model, based on the text description information and the structural information corresponding to the first noise sample, removing the noise of the first output sample corresponding to the first noise sample to obtain a second intermediate sample corresponding to the first noise sample; and updating the features of the second intermediate sample corresponding to the first noise sample to obtain a second output sample corresponding to the first noise sample; updating the second output sample corresponding to the third noise sample based on the second output sample corresponding to the first noise sample and the second output sample corresponding to the fourth noise sample among the plurality of noise samples; The fourth noise sample is used to generate a fourth video frame in the target video, the first video frame, the third video frame and the fourth video frame are sequentially adjacent in the target video, and the second time step is the next time step of the first time step.

5. The method according to any one of claims 1 to 4, characterized in that The target video includes a plurality of segments connected in sequence, each of the segments includes a key frame and a non-key frame, and the first video frame and the other video frames are key frames of different segments respectively; After updating the feature of the first intermediate sample corresponding to the first noise sample to obtain the first output sample corresponding to the first noise sample, the method further includes: Based on the attention mechanism, a feature of the first output sample corresponding to the first noise sample and a feature of the first intermediate sample corresponding to the sixth noise sample are fused with a feature of the first intermediate sample corresponding to the fifth noise sample to obtain a first output sample of the fifth noise sample; The fifth noise sample and the sixth noise sample are respectively used to generate different non-key frames in the segment where the first video frame is located.

6. A video generating device, characterized in that: The device comprises: An acquisition module is used to acquire text description information and multiple structural information of a target video, wherein the multiple structural information corresponds to multiple noise samples one by one; wherein each of the noise samples is used to generate a video frame in the target video, and the structural information corresponding to the noise sample is used to indicate the structure of an object in the video frame; and in a first time step, acquire a first noise sample from the multiple noise samples, wherein the first noise sample is used to generate a first video frame in the target video; a denoising module, configured to use a pre-trained diffusion model to remove noise from the first noise sample based on the text description information and the structural information corresponding to the first noise sample, so as to obtain a first intermediate sample corresponding to the first noise sample; An updating module, configured to update a feature of a first intermediate sample corresponding to the first noise sample, and obtain a first output sample corresponding to the first noise sample; wherein the first output sample of the first noise sample is used as a noise sample of a next time step of the first time step, or the first video frame; The update module is used to: based on the attention mechanism, fuse the features of the first intermediate sample corresponding to the first noise sample with the features of the first intermediate sample corresponding to the first noise sample; wherein the other noise samples are used to generate other video frames in the target video except the first video frame.

7. The device according to claim 6, characterized in that The first intermediate sample corresponding to the first noise sample includes a first area and a second area, and the updating module is used for: Based on the attention mechanism, features of the second region are fused with features of the first region, and features of the first region are fused with features of the second region.

8. The device according to claim 6 or 7, characterized in that The updating module is used to: update the first output sample corresponding to the first noise sample based on the first output sample corresponding to the second noise sample and the first output sample corresponding to the third noise sample; The second noise sample is used to generate a second video frame in the target video, the third noise sample is used to generate a third video frame in the target video, and the second video frame, the first video frame and the third video frame are sequentially adjacent in the target video.

9. The device according to claim 8, characterized in that The denoising module is used to: in a second time step, using the diffusion model, based on the text description information and the structural information corresponding to the first noise sample, remove the noise of the first output sample corresponding to the first noise sample to obtain a second intermediate sample corresponding to the first noise sample; The updating module is used to: update the characteristics of the second intermediate sample corresponding to the first noise sample to obtain a second output sample corresponding to the first noise sample; The updating module is used to: update the second output sample corresponding to the third noise sample based on the second output sample corresponding to the first noise sample and the second output sample corresponding to the fourth noise sample among the multiple noise samples; The fourth noise sample is used to generate a fourth video frame in the target video, the first video frame, the third video frame and the fourth video frame are sequentially adjacent in the target video, and the second time step is the next time step of the first time step.

10. The device according to any one of claims 6 to 9, characterized in that: The target video includes a plurality of segments connected in sequence, each of the segments includes a key frame and a non-key frame, and the first video frame and the other video frames are key frames of different segments respectively; The updating module is used to: based on the attention mechanism, fuse the features of the first output sample corresponding to the first noise sample and the features of the first intermediate sample corresponding to the sixth noise sample into the features of the first intermediate sample corresponding to the fifth noise sample, so as to obtain the first output sample of the fifth noise sample; The fifth noise sample and the sixth noise sample are respectively used to generate different non-key frames in the segment where the first video frame is located.

11. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 5.

13. A computer program product comprising instructions, characterized in that When the instructions are executed by a computer device cluster, the computer device cluster executes the method according to any one of claims 1 to 5.

Citation Information

Cited By

  • Video generation method and device and electronic equipment

    CN119741221A