Video generation method and device and electronic equipment

Abstract

The invention relates to a video generation method and device and electronic equipment. The method comprises the steps that a to-be-processed video clip is acquired, the video clip comprises P frames of edge detection images and Q frames of placeholder images, P is larger than or equal to 1, and Q is larger than or equal to 0; the condition information is input into a pre-trained video generation model, a target video is generated, the condition information at least comprises a to-be-processed video clip, and the target video comprises visual content corresponding to the P frames of edge detection images; wherein the video generation model comprises a diffusion model and an adaptation module, the diffusion model comprises a plurality of down-sampling hidden layers, the adaptation module comprises a plurality of fine-tuning hidden layers, and a fine-tuning vector output by a single fine-tuning hidden layer is used for being superposed with an output vector of the mapped down-sampling hidden layer to serve as an input vector of the next down-sampling hidden layer. According to the scheme provided by the invention, a high-quality video finished product can be generated by adopting the sparsely distributed edge line diagram, and the creation requirement of a user is met.

Description

Technical Field

[0001] The present application relates to the field of computer vision processing technology, and in particular to a video generation method, device and electronic device. Background Art

[0002] With the rapid development of artificial intelligence technology, AIGC (Artificial Intelligence Generated Content) has been widely used in various fields. Among them, the application of video-generated video is booming.

[0003] However, the current image-generated video models have high requirements for images. If the image content is single or the number of images is small, it is difficult to generate a video with rich content, or the video content is incoherent and unreasonable, which limits the quality of the creator's finished video. Summary of the invention

[0004] In order to solve or partially solve the problems existing in the related art, the present application provides a video generation method, device and electronic device, which can use sparsely distributed edge line maps to generate rich video products to meet the user's creative needs.

[0005] The first aspect of the present application provides a video generation method, which includes:

[0006] Obtain a video clip to be processed, wherein the video clip includes a P frame edge detection map and a Q frame placeholder map, where P≥1 and Q≥0;

[0007] Inputting condition information into a pre-trained video generation model to generate a target video, wherein the condition information at least includes the video segment to be processed, and the target video includes visual content corresponding to the P frame edge detection image;

[0008] Among them, the video generation model includes a diffusion model and an adaptation module, the diffusion model includes multiple layers of downsampling hidden layers, the adaptation module includes multiple layers of fine-tuning hidden layers, and the fine-tuning vector output by a single fine-tuning hidden layer is used to superimpose with the output vector of the mapped downsampling hidden layer to serve as the input vector of the next downsampling hidden layer.

[0009] In some implementations, the number of fine-tuning hidden layers of the adaptation module is less than or equal to the number of downsampling hidden layers of the encoder and corresponds layer by layer; the vector dimension of the fine-tuning hidden layer is the same as the vector dimension of the mapped downsampling hidden layer.

[0010] In some implementation modes, obtaining the video clip to be processed includes:

[0011] Obtain an original video clip or a P frame original sample image; generate an edge detection map from the P frame original image in the original video clip, or generate an edge detection map from the P frame original sample image; generate a video clip to be processed according to the edge detection map of the P frame and the Q frame placeholder map.

[0012] In some implementations, the placeholder image is a solid color image; preferably, the placeholder image is a pure black image.

[0013] In some implementations, the condition information further includes prompt text;

[0014] Before inputting the conditional information into the pre-trained video generation model to generate the target video, the method further includes: obtaining the prompt text based on a custom design and / or according to the visual content of the original video clip or the original sample image for the expected target.

[0015] In some implementations, the video generation model is trained according to the following method:

[0016] Acquire multiple sets of training data; wherein each set of the training data includes a first training video and a second training video and corresponding text annotations, the first training video includes a K-frame edge detection image, the second training video includes a P-frame edge detection image and a Q-frame placeholder image training video and corresponding text annotations, P≥1, Q≥0, P+Q=K;

[0017] In each round of training, the second training video is input into the adaptation module, and the fine-tuning vector output by each fine-tuning hidden layer is saved respectively;

[0018] Inputting random noise, the first training video and the text annotation into the diffusion model to obtain a predicted noise image output by the diffusion model; wherein the output vector of each down-sampling hidden layer in the diffusion model is superimposed with the fine-tuning vector of the mapped fine-tuning hidden layer as the input vector of the next down-sampling hidden layer;

[0019] The network parameters of each layer of the diffusion model are fixed, and the network parameters of each layer of the fine-tuning hidden layer of the adaptation module are adjusted according to the back propagation of the loss function of the diffusion model to obtain a trained video generation model.

[0020] In some embodiments, the method further comprises:

[0021] Before training the adaptation module, the network parameters of each down-sampling hidden layer are used as initialization parameters of each fine-tuning hidden layer of the mapping in the adaptation module.

[0022] A second aspect of the present application provides a video generation device, comprising:

[0023] A video acquisition module is used to acquire a video segment to be processed, wherein the video segment includes a P frame edge detection map and a Q frame placeholder map, where P≥1 and Q≥0;

[0024] A video generation module, configured to input condition information into a pre-trained video generation model to generate a target video, wherein the condition information at least includes the video segment to be processed, and the target video includes visual content corresponding to the P frame edge detection map;

[0025] Among them, the video generation model includes a diffusion model and an adaptation module, the diffusion model includes multiple layers of downsampling hidden layers, the adaptation module includes multiple layers of fine-tuning hidden layers, and the fine-tuning vector output by a single fine-tuning hidden layer is used to superimpose with the output vector of the mapped downsampling hidden layer to serve as the input vector of the next downsampling hidden layer.

[0026] A third aspect of the present application provides an electronic device, including:

[0027] Processor; and

[0028] A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the video generating method as described in the first aspect above.

[0029] A fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor executes the video generation method described in the first aspect above.

[0030] A fifth aspect of the present application provides a computer program product, including a computer program, characterized in that the computer program is used to execute computer program code instructions corresponding to the video generation method described in the first aspect above.

[0031] The technical solution provided by this application may have the following beneficial effects:

[0032] The video generation method of the present application, by using a diffusion model with an adaptation module as a new video generation model, can restore or reconstruct the video content based on a small number of edge detection graphs, accurately capture as many edges as possible in the image, and generate high-quality target videos to meet the user's video production needs.

[0033] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail the exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0035] Figure 1 It is a flowchart of a video generation method provided in an embodiment of the present application;

[0036] Figure 2 is a schematic diagram of the structure of the adaptation module and the diffusion model provided in the embodiment of the present application;

[0037] Figure 3 It is a flowchart of a training method for a video generation model provided in an embodiment of the present application;

[0038] Figure 4 is another flow chart of a video generation method provided by an embodiment of the present application;

[0039] Figure 5 is a schematic diagram of the structure of a video generating device provided in an embodiment of the present application;

[0040] Figure 6 is another structural schematic diagram of the video generating device provided in an embodiment of the present application;

[0041] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0043] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0044] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0045] In the related technology, in the current AIGC field, users are limited to generating videos from some high-quality images, and have high requirements on the quality of input images, which affects the creators' creative needs.

[0046] In response to the above problems, the present application provides a video generation method that can use sparsely distributed edge line graphs to generate high-quality video products to meet user creative needs.

[0047] The technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0048] See also Figure 1 , a video generation method shown in the present application includes:

[0049] S110, obtaining a video segment to be processed, where the video segment includes a P frame edge detection map and a Q frame placeholder map; wherein P≥1, Q≥0.

[0050] In the present application, the edge detection map can be an image obtained by processing the original image through an edge detection algorithm (such as the Canny algorithm); in a video clip, the frame numbers corresponding to multiple frames of edge detection maps can be continuous or intermittent, that is, the arrangement positions of multiple frames of edge detection maps in the video clip can be customized. Except for the edge detection map, other images in the video clip are placeholders. Among them, the edge detection map includes edge information in the original image, which is convenient for accurately identifying objects in the image. Therefore, on the basis of accurately capturing as many edges as possible in the image, more accurate information can be given to the video generation model in the future, and a better target video can be generated. A placeholder map refers to an image that does not contain visual content. On the basis of combining the edge detection map, the subsequent video generation model can be created based on edge information to generate new images related to edge information to enrich the visual content. In some embodiments, each frame of the placeholder map can be represented by a solid color map. Different placeholder maps can be represented by different colors, or they can be represented by the same color. Preferably, the placeholder map can be a pure black map. It can be understood that, taking the grayscale value as an example, the grayscale value of each pixel in a pure color image is the same; for example, the grayscale value of each pixel in a pure black image is 0.

[0051] S120, inputting condition information into a pre-trained video generation model to generate a target video; the condition information at least includes the video segment to be processed, and the target video includes visual content corresponding to the P frame edge detection image.

[0052] The video generation model includes a diffusion model and an adaptation module. The diffusion model includes multiple layers of downsampling hidden layers and multiple layers of upsampling hidden layers. The adaptation module has multiple layers of fine-tuning hidden layers, and the multiple layers of fine-tuning hidden layers of the adaptation module are mapped one-to-one with the multiple layers of downsampling hidden layers of the diffusion model, and the structure of each layer of fine-tuning hidden layer is the same as that of the corresponding downsampling hidden layer. The fine-tuning vector output by the single-layer fine-tuning hidden layer of the adaptation module is used to superimpose the output vector of the mapped downsampling hidden layer of the diffusion model as the input vector of the downsampling hidden layer of the next layer.

[0053] The pre-trained video generation model is obtained by joint training of the diffusion model and the adaptation module. Optionally, in the process of joint training, first: obtain the pre-trained diffusion model and the adaptation module to be trained, wherein the pre-trained diffusion model has the ability to generate videos, but the effect of its video generation is poor for specific scenes, and the network parameter magnitude of the pre-trained diffusion model is relatively large, such as billions or tens of billions. If the parameters of the pre-trained diffusion model are directly retrained and adjusted for specific scenes, it will not only affect the versatility of the pre-trained diffusion model, but also require very large training resources and a long cycle. The adaptation module includes a multi-layer fine-tuning hidden layer, and the diffusion model includes a multi-layer downsampling hidden layer (i.e., input layer) and a multi-layer upsampling hidden layer (i.e., output layer). In this embodiment, the multi-layer fine-tuning hidden layer of the adaptation module to be trained has the same layer structure as the multi-layer downsampling hidden layer of the diffusion model, that is, the number of layers of the network layer of the adaptation module is the same as the number of layers of the network layer of the input layer of the diffusion model, and the network layer of each adaptation module has the same network parameters as the network layer of the input layer of the corresponding diffusion model. Secondly: fix the network parameters of each layer of the diffusion model, and adjust the network parameters of each fine-tuning hidden layer of the adaptation module according to the back propagation of the loss function of the diffusion model, so as to obtain the pre-trained video generation model.

[0054] In an embodiment of the present invention, a pre-trained video generation model is obtained by jointly training a diffusion model and an adaptation module, and the parameters of the diffusion model do not need to be directly adjusted, which can not only reduce the number of parameters required for fine-tuning, but also reduce the demand for computing resources during training, thereby speeding up the training speed. At the same time, when the diffusion model performs a specific task, the vector of the input layer of the diffusion model is fine-tuned through the pre-trained adaptation module, thereby achieving the specific task that can be completed efficiently and accurately without the need for large-scale overall training of the diffusion model. It can be seen that the pre-trained video generation model is obtained by jointly training the diffusion model and the adaptation module, which realizes the rapid generation of high-quality target videos with the visual content of the acquired video clips to be processed based on AIGC technology, and does not affect the versatility of the diffusion model.

[0055] In an embodiment of the present invention, the diffusion model is a fusion architecture of Transformer and Diffusion, that is, after the input data is mapped to the latent space expression by the encoder, the latent space expression of the target output corresponding to the input data is obtained through the process of denoising and denoising, and the latent space expression of the target output is mapped to the corresponding data space by the decoder, and finally the target output is obtained. Specifically, the Transformer model architecture is a deep learning architecture that relies on the self-attention mechanism, and is essentially composed of an encoder and a decoder.

[0056] Furthermore, the diffusion model is a generative model based on a deep learning algorithm, which generates high-quality content through a process of gradual denoising and denoising, performs well in generating images, videos, audio and other high-dimensional data, and has strong performance in generating tasks. In this application, the diffusion model is a pre-trained model. After the above-mentioned encoder maps the edge detection map to the latent space expression, under the guidance of the latent space expression of the edge detection map, the random noise is denoised for T time steps, that is, downsampling and upsampling are performed at each time step to obtain the latent space expression of the target video, and then the latent space expression of the target video is further mapped to the pixel space through the above-mentioned decoder to obtain the target video. Among them, the latent space expression of the edge detection map is the latent space expression obtained after encoding the edge detection map using the encoder. The encoder is responsible for mapping the data from the original data space to the latent space (Latent Space) to obtain an implicit and continuous representation, which can make the model calculation more efficient; and the decoder can reconstruct the video from this latent space expression and map the generation result of the diffusion model from the latent space to the pixel space. In some embodiments, the encoder and decoder used in the diffusion model of the present application can both be variational autoencoders (VAE). The latent space of VAE is continuous and suitable for generating continuous data, that is, the generated target videos or images are correlated.

[0057] For example, Figure 2As shown, the core module in the diffusion model is the U-Net network. U-Net is a structure of a downsampling module-upsampling module with jump connections, which is used to denoise random noise and realize the generation from noise to video or image. Among them, the downsampling module is used to gradually downsample the input noise, extract features and reduce the spatial resolution of the image. The upsampling module gradually restores the spatial resolution of the image through deconvolution (or transposed convolution) and upsampling operations; through jump connections, the upsampling module can combine low-level features (such as edge, texture and other detail information) in the downsampling module with high-level features (such as semantic information of prompt text), and the fusion of features helps to reconstruct the details of the image; finally, the target image with the same resolution as the input image is output. In this application, based on the multi-layer downsampling hidden layer structure in the downsampling module of the diffusion model, an independent adaptation module is designed to fine-tune the vector generated by the downsampling process. That is to say, the adaptation module adapts to new tasks by inserting additional parameters into the pre-trained diffusion model, rather than directly adjusting the parameters of the diffusion model (the parameters of the diffusion model are in the order of billions or tens of billions). This can not only reduce the number of parameters required for fine-tuning, but also reduce the demand for computing resources during training, thereby speeding up training. At the same time, when the diffusion model performs a specific task, the output vectors of each layer of the downsampled hidden layer are fine-tuned through the trained adaptation module, so that the diffusion model does not need to be trained on a large scale as a whole, and the specific task can be completed efficiently and accurately. It can be seen that through the combination of the adaptation module and the diffusion model, it is possible to quickly generate high-quality target videos based on AIGC technology through video clips including edge detection maps, without affecting the versatility of the diffusion model.

[0058] See also Figure 2 , the adaptation module has multiple layers of fine-tuning hidden layers arranged in sequence, and the diffusion model includes multiple layers of downsampling hidden layers arranged in sequence. In some embodiments, the number of layers of the fine-tuning hidden layers of the adaptation module is less than or equal to the number of layers of the downsampling hidden layers and corresponds layer by layer. For example, the adaptation module has M layers of fine-tuning hidden layers, and the downsampling module has N layers of downsampling hidden layers, then 1≤M≤N, and M and N are both positive integers. Specifically, when M is equal to N, each layer of fine-tuning hidden layer is mapped one-to-one with the downsampling hidden layer of the same level. When M is less than N, each layer of fine-tuning hidden layer can be mapped one-to-one with the downsampling hidden layer of the same level, or it can be mapped one-to-one layer by layer with downsampling hidden layers of different levels, such as mapping to downsampling hidden layers corresponding to odd layers, even layers, specified layers or random layers. Such a design allows the input vector of at least one downsampled hidden layer to be fine-tuned by superimposing a fine-tuning vector, and then changing the corresponding output vector, so that the general diffusion model is affected by the adaptation module of a specific task and tends to process the specific task to achieve the expected goal.

[0059] Furthermore, in some implementations, the vector dimension of the fine-tuning hidden layer is the same as the vector dimension of the down-sampling hidden layer of the corresponding level. Such a design facilitates the superposition of the fine-tuning vector output by the fine-tuning hidden layer and the output vector of the corresponding down-sampling hidden layer in the same dimension, so as to achieve comprehensive fine-tuning of the down-sampling hidden layer.

[0060] It can be understood that after the conditional information is input into the video generation model, the latent space expression corresponding to these conditional information is obtained through the processing of the encoder. The latent space expression of the video clip to be processed in the conditional information is input into the adaptation module, and each fine-tuning hidden layer of the adaptation module gradually completes the downsampling of the latent space expression; the fine-tuning vector output by each fine-tuning hidden layer is used as the input vector of the next fine-tuning hidden layer, and the fine-tuning vector output by each fine-tuning hidden layer is saved. Correspondingly, the random noise is input into the diffusion model for denoising, and down-sampled layer by layer through each down-sampling hidden layer of the diffusion model; in the down-sampling process, the output vector of each down-sampling hidden layer is superimposed with the mapped fine-tuning vector to form the input vector of the next down-sampling hidden layer. And so on, until the output vector of the last down-sampling hidden layer is superimposed with the corresponding fine-tuning vector to form the final output vector of the down-sampling module. The final output vector of the down-sampling module is used as the input vector of the up-sampling module for up-sampling. Optionally, the output vector of the down-sampling hidden layer and the corresponding fine-tuning vector can be directly superimposed or weighted superimposed.

[0061] It can be understood that the above-mentioned downsampling module and upsampling module perform one round of calculation as a noise prediction for one time step, that is, a denoising process for one time step, and the diffusion model performs T rounds of iterative denoising to obtain the final target video. After the upsampling module outputs the output vector of the first round, it is input into the downsampling module again as the initial input vector of the second round. After T rounds of loop execution according to the above process, the upsampling module outputs the output vector of the Tth round to input the decoder to obtain the target video. In each round of iteration of the diffusion model, each downsampling hidden layer of the downsampling module needs to be superimposed with the fine-tuning vector of the fine-tuning hidden layer mapped by the above-mentioned adaptation module. The decoder can reconstruct the video from the latent space expression, map the generation result of the diffusion model from the latent space to the pixel space, and generate the target video.

[0062] It can be understood that the conditional information at least includes the video clip to be processed. Based on this, the target video generated by the video generation model includes the visual content corresponding to the above-mentioned P frame edge detection map. That is to say, the conditional information includes the reference content expected by the user (the video clip includes a sparse edge detection map). Based on the guidance of the video generation model by the reference content, a high-quality target video that meets the user's expectations can be generated, thereby improving the applicability of users using the edge detection map corresponding to any video frame in the video clip to generate high-quality videos, expanding the application field of AIGC technology, and improving the interest of AIGC technology application. Specifically, based on the adjacently arranged edge detection map, the Q frame placeholder map can generate visual content that is continuous with the adjacent image action, so that the overall content of the target video is complete and reasonable. On the basis of the visual content corresponding to the edge detection map, the video generation model, based on the learned knowledge, enables the generated target video to also include other matching content, such as color, background image, etc., so that the target video content is more colorful.

[0063] From this example, we can see that the video generation method of the present application, by using the diffusion model with the adaptation module as a new video generation model, can restore or reconstruct the video content based on a small amount of edge detection graphs in combination with the input conditional information. Such a design can obtain a high-quality target video with the same duration and number of frames as the video clip, meeting the user's video production needs.

[0064] See also Figure 3 An embodiment of the present application also provides a training method for the video generation model of the present application, wherein the video generation model to be trained is composed of a pre-trained diffusion model and an adaptation module to be trained, wherein the pre-trained diffusion model can realize the function of video generation, but the application scenario is relatively limited, and it is impossible to generate a high-quality video that meets the user's expectations according to an edge detection map or a video. The video generation model of the present application can be obtained by training according to the following method.

[0065] S210, obtaining multiple groups of training data; wherein each group of training data includes a first training video and a corresponding second training video and text annotations, the first training video includes K frame edge detection images, the second training video includes P frame edge detection images and Q frame placeholder image training videos and corresponding text annotations, P≥1, Q≥0, P+Q=K.

[0066] In the present application, the frame rate and duration of the target video generated by the video generation model can be preset, the total number of frames K of the target video can be determined, and then the total number of frames of the first training video in each group of training data can be determined to be K.

[0067] In the same set of training data, each frame image of the first training video is an edge detection image, and the second training video is a video obtained by replacing the Q frame placeholder image based on the first training video. The number of image frames of the two is the same, but the number of frames of the edge detection images of the two may be the same or different, that is, P≤K.

[0068] Furthermore, different second training videos may contain edge detection images of the same or different numbers of frames, that is, the number of frames P of edge detection images in each second training video can be customized. In other words, a second training video may contain multiple frames of sparsely dispersed edge detection images, and the remaining images may be replaced by placeholder images; each frame image of the second training video may also be entirely edge detection images.

[0069] In order to obtain high-quality training videos to improve the training efficiency of the model, sample videos with clear images and rich content are screened. The sample videos have the same frame rate and duration as the first training video, and thus have the same number of frames. Also, after converting each frame of the sample image of the sample video into an edge detection map through an edge detection algorithm, the first training video is obtained, and then P frame edge detection maps are randomly selected from the first training video to form the second training video. Optionally, the P frame edge detection maps can be arranged according to the corresponding frame sequence in the first training video, and the images of the remaining frame sequences are replaced by placeholder maps.

[0070] Optionally, based on the differences in the edge detection images randomly selected each time in the same first training video (such as different numbers or different frame sequences), multiple different second training videos can be generated to form corresponding multiple sets of training data. It can be understood that based on a sample video having a corresponding text annotation for describing the video content, when multiple sets of training data have the same first training video and different second training videos, each set of training data can use the same text annotation, so that the adaptation module can learn the association between the visual content of different edge detection images based on the same prompt words during the training process, and then restore the video content more accurately.

[0071] In some other implementations, K unrelated sample frames may be selected to form corresponding edge detection images, and the K frame edge detection images may be used to form a first training video, and the P frame edge detection images and Q frame placeholder images therein may be used to form a second training video. It is understandable that for the same first training video, multiple frames of sample images do not need to come from the same sample video, for example, they may be randomly selected from different sample videos, or existing images may be selected, and then these unrelated sample images may be converted into edge detection images respectively, and then orderly or randomly combined in the same first training video. Accordingly, the text annotation corresponding to the first training video is set according to the currently selected K frame sample images. Such a design, by forming a training video with unrelated edge detection images into a content, enables the adaptation module to learn the association between edge detection images with large content spans during the training process, and then, under the guidance of the prompt words, it is possible to more accurately reconstruct the target video.

[0072] S220, in each round of training, input the training data into the adaptation module of the video generation model, and save the fine-tuning vectors output by each fine-tuning hidden layer respectively.

[0073] In this step, the model architecture of the adaptation module is pre-set. Figure 2 As shown, a trained diffusion model is selected, which has a corresponding network structure U-Net, and the network structure of the adaptation module is set to be consistent with the network structure of the downsampling module in the diffusion model. That is, the number of layers of the fine-tuning hidden layer of the adaptation module is consistent with the number of layers of the downsampling hidden layer, and the vectors of the fine-tuning hidden layer and the downsampling hidden layer have the same dimension. It should be noted that, in other embodiments, the number of layers of the fine-tuning hidden layer of the trained adaptation module can also be less than the number of layers of the downsampling hidden layer, and there is no limitation here.

[0074] In some implementations, the network parameters of the adaptation module are initialized before training the adaptation module. Specifically, the network parameters of each down-sampling hidden layer in the pre-trained diffusion model are used as the initialization parameters of each fine-tuning hidden layer of the mapping of the adaptation module.

[0075] After initializing the adaptation module, the training of the adaptation module is officially started. After encoding and mapping the edge detection image and placeholder map of each frame of the second training video to the latent space expression in the encoder, the latent space expression is input into the adaptation module, and each fine-tuning hidden layer outputs the corresponding fine-tuning vector and saves it.

[0076] It should be noted that this step S220 can be performed synchronously and alternately with S230, that is, according to the number of fine-tuning hidden layers, each time S220 is executed, S230 is executed once, S220 is executed again, and S230 is executed once, until the last fine-tuning hidden layer outputs a fine-tuning vector, that is, S220 is terminated. Alternatively, S220 is executed first, and after the fine-tuning vectors output by all fine-tuning hidden layers are saved, S230 is executed.

[0077] S230, random noise, the first training video and text annotations are input into the diffusion model of the video generation model to obtain a predicted noise image output by the diffusion model; wherein the output vector of each downsampling hidden layer of the diffusion model is superimposed with the fine-tuning vector of the mapped fine-tuning hidden layer as the input vector of the next downsampling hidden layer.

[0078] Optionally, the diffusion model and the adaptation module may share an encoder, or may use independent encoders. In this step, each frame of the edge detection image of the second training data in S210 is randomly denoised and input into the diffusion model together with the text annotation. Referring to the operating principle of the diffusion model in step S120, the random noise is denoised for T time steps through the conditional guidance of the edge detection image to obtain a predicted noise image.

[0079] Taking the case where the number of fine-tuning hidden layers is the same as the number of down-sampling hidden layers, after the output vector of the first down-sampling hidden layer is obtained, it is added to the fine-tuning vector of the output of the first fine-tuning hidden layer obtained in S220 to obtain the input vector of the second down-sampling hidden layer. After the output vector of the second down-sampling hidden layer is obtained, it is added to the fine-tuning vector of the output of the second fine-tuning hidden layer to obtain the input vector of the third down-sampling hidden layer, and so on, until the output vector of the last down-sampling hidden layer is added to the fine-tuning vector of the output of the last fine-tuning hidden layer, and enters the up-sampling module of the diffusion model through forward propagation for up-sampling, and finally enters the decoder to output the predicted noise image corresponding to each frame of the input image. The specific principle can be referred to the introduction of the above diffusion model, which will not be repeated here.

[0080] S240, fixing the network parameters of each layer of the diffusion model, and adjusting the network parameters of each layer of the adaptation module and fine-tuning the hidden layer according to the back propagation of the loss function of the diffusion model to obtain a trained video generation model.

[0081] In the present application, the training process is to iteratively adjust the network parameters of the adaptation module, and the network parameters of each layer of the diffusion model are fixed. Based on this, after calculating the noise loss function of the true value pixels of the edge detection image corresponding to the predicted noise image and the first training video, the network parameters of each fine-tuning hidden layer of the adaptation module are adjusted by back propagation. In this way, according to the loop iteration from S220 to S240 until the loss function converges, a trained adaptation module can be obtained, and each fine-tuning hidden layer has a trained network parameter.

[0082] From this example, it can be seen that the training method of the present application can compose a first training video with content-related or unrelated edge detection images, and obtain a second training video with sparse or continuous expression of visual content. Based on text annotation to provide semantic features, an adaptation module is connected to the downsampling module of the pre-trained diffusion model to realize training of the adaptation module, thereby efficiently and low-costly obtaining a video generation model that can restore or reconstruct video content.

[0083] See also Figure 4 According to the trained video generation model of the above embodiment, the video generation method of the present application will be further introduced in combination with the embodiment below, which includes:

[0084] S310, generating an edge detection map from the P frame original image in the original video segment, or generating an edge detection map from the P frame original sample image, and generating a video segment to be processed according to the P frame edge detection map and the Q frame placeholder map, wherein P≥1, Q≥0.

[0085] In the present application, the frame rate and duration of the video clip to be processed can be determined according to the frame rate and duration of the target video. The video clip to be processed includes K frames of images, P+Q=K, and P, Q and K are all natural numbers. Among them, the number P of edge detection images can be selected from 1 to K, that is, the number of edge detection images is at least 1 frame, or each frame of the video clip is an edge detection image. On the contrary, according to the total number of frames K of the video clip, the number of placeholder images Q is 0 to (K-1). It can be understood that when the number of edge detection images is greater than 1 frame, according to the total number of frames of the video clip, multiple frames of edge detection images can be arranged continuously or at intervals.

[0086] For some original video clips, there are situations where the resolution is low or part of the image content is damaged. When the intended goal is to restore the original video content, in some implementations, the original video clip is obtained; when the total number of frames of the original video clip is equal to the number of frames of the video clip to be processed, the P frame original image in the original video clip can be converted into a corresponding edge detection map, and the remaining original images are replaced with placeholder images to generate the video clip to be processed. For example, a higher quality original image in the original video clip can be selected and converted into an edge detection map, thereby improving the degree of restoration of the video content. It can be understood that each frame edge detection map can be arranged according to the frame number of the corresponding original image to help restore the visual content. Of course, in other embodiments, the edge detection maps of each frame can also be arranged in random order and then used to reconstruct the visual content.

[0087] In some implementations, when the total number of frames of the original video clip is less than the number of frames of the video clip to be processed, at least P frames of the original image are converted into edge detection images, the missing image frames are replaced with placeholder images, or the edge detection images are copied for placeholder, or other edge detection images with similar content are used for placeholder. In some implementations, when the total number of frames of the original video clip is greater than the total number of frames of the video clip to be processed, the original video clip can be edited to match the total number of frames of the video clip.

[0088] When the user's expected goal is to reconstruct the content of the original video clip, the above-mentioned processing method of the original video clip is also applicable and will not be described in detail here.

[0089] When the user's intended goal is to reconstruct the content of the original video clip, the user can also generate a video clip to be processed based on the existing P-frame original sample image. In some other implementations, the P-frame original sample image is converted into an edge detection map; the P-frame original sample image and the Q-frame placeholder map are spliced ​​to form the video clip to be processed. In other words, when the original video clip is missing, the user can select some original sample images to generate an edge detection map to achieve free creation of video content, such as based on a variety of visual content of unrelated original sample images, so that the visual content of the generated target video can be richer. For example, based on the content of related original sample images, the generated target video can restore the dynamic scene when the image was shot.

[0090] S320, inputting the video clip and the prompt text into a pre-trained video generation model to generate a target video, wherein the video generation model includes a diffusion model and an adaptation module.

[0091] In this embodiment, the condition information also includes a prompt text. The prompt text includes at least the attribute information of the main object in the target video, for example, it may include the type of the main object and other attribute information of the main object. The type of the main object may include: the type of person, the type of animal or other object (for example, an object that needs to be moved in a movie or game scene). Other attribute information of the main object may include the color information of the main object, the structure information (for example, when the main object is a person or an animal, it may include the body proportions of the main object) and other information.

[0092] As a general video generation model, the diffusion model has a relatively strong ability in text understanding. In this embodiment, by taking the prompt text as the input of the diffusion model, it can be ensured that the generated target video better meets the user's expectations.

[0093] For the expected goal, the prompt text is obtained based on the custom design and / or according to the visual content of the original video clip or the original sample image. When the expected goal is to restore the video content, the prompt text needs to be as consistent as possible with the visual content of the original video clip or the original sample image. When the expected goal is to reconstruct the video content, the prompt text can be different from the visual content of the original video clip or the original sample image.

[0094] In this embodiment, the video generation model includes a diffusion model and an adaptation module, and the diffusion model and the adaptation module may have the same encoder. The downsampling hidden layer of the diffusion model and the fine-tuning hidden layer of the adaptation module have the same number of layers and the same dimension.

[0095] After random noise is added to each frame of the video clip, it is input into the video generation model together with the prompt text. According to the latent space expression corresponding to each frame of the encoded input image, each fine-tuning hidden layer in the adaptation module outputs the corresponding fine-tuning vector Δd one by one; for example, the first fine-tuning hidden layer outputs the fine-tuning vector Δd1, the second fine-tuning hidden layer outputs the fine-tuning vector Δd2, and the nth fine-tuning hidden layer outputs the fine-tuning vector Δd n .

[0096] Correspondingly, the output vector of the first downsampling hidden layer in the diffusion model is d1. After d1 is superimposed with Δd1, (d1+Δd1) is used as the input vector of the second downsampling hidden layer. The output vector of the second downsampling hidden layer is d2. After d2 is superimposed with Δd2, (d2+Δd2) is used as the input vector of the third downsampling hidden layer. Similarly, the output vector of the last downsampling hidden layer is d n With Δd nAfter superposition, the final output vector is formed to be input into the upsampling module for upsampling. The downsampling module and the upsampling module perform denoising according to the prompt information of the prompt text to obtain the latent space expression that conforms to the visual content of the target video. After T rounds of iterative denoising, it is input into the decoder to obtain the target video.

[0097] In summary, in the video generation method of the present application, the user can select diversified sample source data according to the expected goal to obtain the video clip to be processed. The video generation model can restore or reconstruct the video content based on the edge detection image and the prompt text. With the assistance of the pre-trained adaptation module and the prompt text, the visual content of each frame image of the target video is close to the user's expectations, thereby meeting the user's creative needs.

[0098] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides a video generation device and corresponding embodiments.

[0099] Figure 5 It is a schematic diagram of the structure of the video generating device shown in this application.

[0100] See also Figure 5 , the video generation device shown in the present application includes a video acquisition module 510 and a video generation module 520; wherein:

[0101] The video acquisition module 510 is used to acquire a video segment to be processed, where the video segment includes a P frame edge detection map and a Q frame placeholder map, where P≥1 and Q≥0.

[0102] The video generation module 520 is used to input condition information into a pre-trained video generation model to generate a target video, where the condition information at least includes a video segment to be processed, and the target video includes visual content corresponding to the P frame edge detection image.

[0103] Among them, the video generation model includes a diffusion model and an adaptation module. The diffusion model includes multiple layers of downsampling hidden layers. The adaptation module includes multiple layers of fine-tuning hidden layers. The fine-tuning vector output by a single fine-tuning hidden layer is used to superimpose the output vector of the mapped downsampling hidden layer to serve as the input vector of the next downsampling hidden layer.

[0104] See also Figure 6In some implementations, the video acquisition module 510 includes a data acquisition module 511 and an image processing module 512. The data acquisition module 511 is used to acquire the original video clip or the P frame original sample image. The image processing module 521 is used to generate an edge detection map from the P frame original image in the original video clip, or to generate an edge detection map from the P frame original sample image; and to generate a video clip to be processed according to the P frame edge detection map and the Q frame placeholder map. The video acquisition module 510 also includes a text acquisition module 513, which is used to acquire a prompt text corresponding to the video clip to be processed.

[0105] From this example, it can be seen that the video generation device of the present application, through a trained video generation model, can generate a target video based on a sparsely distributed / continuously distributed edge detection map, wherein an adaptation module is used to fine-tune the input vector of the downsampling hidden layer of the diffusion model, and combined with the semantic features in the prompt text, the generated target video can achieve the effect of restoring or reconstructing the content, thereby meeting the user's creative needs.

[0106] An embodiment of the present application also shows a training device for a video generation model, which includes:

[0107] The sample acquisition module is used to obtain multiple groups of training data; each group of training data includes a first training video and a second training video and corresponding text annotations, the first training video includes a K-frame edge detection image, the second training video includes a P-frame edge detection image and a Q-frame placeholder image, P≥1, Q≥0, P+Q=K.

[0108] The fine-tuning vector module is used to input the second training video into the adaptation module in each round of training, and respectively save the fine-tuning vector output by each fine-tuning hidden layer.

[0109] The model training module is used to input random noise, a first training video and text annotations into a diffusion model to obtain a predicted noise image output by the diffusion model; wherein the output vector of each down-sampling hidden layer of the diffusion model is superimposed with the fine-tuning vector of the mapped fine-tuning hidden layer as the input vector of the next down-sampling hidden layer.

[0110] The parameter iteration module is used to fix the network parameters of each layer of the diffusion model, and adjust the network parameters of each layer of the adaptation module and fine-tune the hidden layer according to the back propagation of the loss function of the diffusion model to obtain a trained video generation model.

[0111] The training device of the video generation model of the present application adapts to specific downstream tasks by inserting a small, trainable adaptation module into a pre-trained diffusion model without fine-tuning the entire diffusion model, allowing the diffusion model to adapt to new tasks by adjusting a small part of parameters while keeping the original pre-trained parameters unchanged, thereby improving the parameter training efficiency and training speed.

[0112] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0113] Figure 7 It is a schematic diagram of the structure of the electronic device shown in this application.

[0114] See also Figure 7 , the electronic device 1000 includes a memory 1010 and a processor 1020 .

[0115] The processor 1020 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0116] The memory 1010 may include various types of storage units, such as system memory, read-only memory (ROP), and permanent storage. Among them, the ROP may store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage may be a readable and writable storage device. The permanent storage may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAP, SRAP, SDRAP, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROP, double-layer DVD-ROP), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, PiQ SD card, Picro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0117] The memory 1010 stores executable codes, and when the executable codes are processed by the processor 1020 , the processor 1020 can execute part or all of the methods described above.

[0118] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0119] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0120] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A video generation method, characterized in that: include: Obtain a video clip to be processed, wherein the video clip includes a P frame edge detection map and a Q frame placeholder map, where P≥1 and Q≥0; Inputting condition information into a pre-trained video generation model to generate a target video, wherein the condition information at least includes the video segment to be processed, and the target video includes visual content corresponding to the P frame edge detection image; Among them, the video generation model includes a diffusion model and an adaptation module, the diffusion model includes multiple layers of downsampling hidden layers, the adaptation module includes multiple layers of fine-tuning hidden layers, and the fine-tuning vector output by a single fine-tuning hidden layer is used to superimpose with the output vector of the mapped downsampling hidden layer to serve as the input vector of the next downsampling hidden layer.

2. The method according to claim 1, characterized in that: The number of layers of the fine-tuning hidden layer of the adaptation module is less than or equal to the number of layers of the downsampling hidden layer of the diffusion model; The vector dimension of the fine-tuning hidden layer is the same as the vector dimension of the mapped down-sampling hidden layer.

3. The method according to claim 1, characterized in that The step of obtaining the video clip to be processed includes: Get the original video clip or P frame original sample image; Generate an edge detection map from the original image of the P frame in the original video clip, or generate an edge detection map from the original sample image of the P frame; A video segment to be processed is generated according to the edge detection map of the P frame and the placeholder map of the Q frame.

4. The method according to claim 1, characterized in that: The placeholder image is a solid color image; preferably, the placeholder image is a pure black image.

5. The method according to claim 1 or 3, characterized in that: The condition information also includes prompt text; Before inputting the conditional information into the pre-trained video generation model to generate the target video, it also includes: The prompt text is obtained based on a custom design and / or according to the visual content of the original video clip or the original sample image for the expected purpose.

6. The method according to claim 1, characterized in that The video generation model is trained according to the following method: Acquire multiple sets of training data; wherein each set of the training data includes a first training video and a second training video and corresponding text annotations, the first training video includes a K-frame edge detection image, the second training video includes a P-frame edge detection image and a Q-frame placeholder image training video and corresponding text annotations, P≥1, Q≥0, P+Q=K; In each round of training, the second training video is input into the adaptation module, and the fine-tuning vector output by each fine-tuning hidden layer is saved respectively; Inputting random noise, the first training video and the text annotation into the diffusion model to obtain a predicted noise image output by the diffusion model; wherein the output vector of each down-sampling hidden layer in the diffusion model is superimposed with the fine-tuning vector of the mapped fine-tuning hidden layer as the input vector of the next down-sampling hidden layer; The network parameters of each layer of the diffusion model are fixed, and the network parameters of each layer of the fine-tuning hidden layer of the adaptation module are adjusted according to the back propagation of the loss function of the diffusion model to obtain a trained video generation model.

7. The method according to claim 6, characterized in that The method further comprises: Before training the adaptation module, the network parameters of each down-sampling hidden layer are used as initialization parameters of each fine-tuning hidden layer of the mapping in the adaptation module.

8. A video generating device, characterized in that: include: A video acquisition module is used to acquire a video segment to be processed, wherein the video segment includes a P frame edge detection map and a Q frame placeholder map, where P≥1 and Q≥0; A video generation module, configured to input condition information into a pre-trained video generation model to generate a target video, wherein the condition information at least includes the video segment to be processed, and the target video includes visual content corresponding to the P frame edge detection map; Among them, the video generation model includes a diffusion model and an adaptation module, the diffusion model includes multiple layers of downsampling hidden layers, the adaptation module includes multiple layers of fine-tuning hidden layers, and the fine-tuning vector output by a single fine-tuning hidden layer is used to superimpose with the output vector of the mapped downsampling hidden layer to serve as the input vector of the next downsampling hidden layer.

9. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the video generating method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the video generation method according to any one of claims 1 to 7.

Images