Video generation method, device, electronic device and computer-readable storage medium

Abstract

The disclosed embodiment discloses a method, device, electronic device and computer-readable storage medium for video generation, wherein the method comprises: obtaining a first image set of an input video, wherein the first image set comprises graffiti images of a first number of video frames of the input video and all-black images of a second number of video frames; inputting preset Gaussian noise, target video prompt words and the first image set into a pre-trained video generation model to generate a target video; wherein the pre-trained video generation model is jointly trained by a preset Vincent video model and a preset adaptation model. The process of automatically converting relatively rough line drawings such as graffiti images into animated videos is realized, helping different users to achieve their video generation needs and improving the user's interactive experience.

Description

Technical Field

[0001] The present disclosure relates to video processing technology, and in particular to a method, device, electronic device and computer-readable storage medium for generating a video. Background Art

[0002] Recently, due to the rapid development of video generation model technology, video generation models have shined in the field of AIGC (Artificial Intelligence Generated Content) technology. With the continuous advancement of technology, the application prospects of image-generated video technology in the field of AIGC technology are very broad, and may play an important role in many fields such as film and television production, advertising marketing, education, and entertainment games, and promote the innovation and development of related industries. Especially in the fields of education and entertainment games, innovative support for image-generated video technology is urgently needed. For example, in the field of education, some hand-drawn pictures (or graffiti, etc.) can be obtained through models. Video can be used to teach painting or design skills. In teaching, students can see the process of converting their own graffiti into animation, which helps students cultivate creativity and stimulate imagination; for example, in the field of entertainment games, the game environment can be generated based on the player's graffiti to enhance the interactive experience of the game. Summary of the invention

[0003] In order to solve the technical problems in the related art, the embodiments of the present disclosure provide a method, an apparatus, an electronic device and a computer-readable storage medium for generating a video.

[0004] A first aspect of the present disclosure provides a method for generating a video, the method comprising:

[0005] Acquire a first image set of an input video, wherein the first image set includes scribble images of a first number of video frames of the input video and completely black images of a second number of video frames;

[0006] Inputting preset Gaussian noise, target video prompt words and the first image set into a pre-trained video generation model to generate a target video;

[0007] The pre-trained video generation model is jointly trained by a preset video model and a preset adaptation model.

[0008] As an optional embodiment of the present disclosure, the step of obtaining a first image set of an input video includes:

[0009] Acquire multiple video frames of the input video according to a preset processing method;

[0010] Extracting a graffiti image of the video frame from each of the video frames to obtain a graffiti image set of the input video;

[0011] The target graffiti image in the graffiti image set is replaced with a completely black image to obtain the first image set.

[0012] As an optional embodiment of the present disclosure, replacing the target graffiti image in the graffiti image set with a completely black image to obtain the first image set includes:

[0013] Determine the total number of video frames of the input video;

[0014] Based on the total number of video frames of the input video, a preset proportion of graffiti images in the graffiti image set are determined as target graffiti images, so as to replace the target graffiti images with all-black images to obtain the first image set.

[0015] As an optional embodiment of the present disclosure, determining the total number of video frames of the input video includes:

[0016] Obtain the total duration and frame rate of the input video;

[0017] The total number of video frames of the input video is calculated according to the total duration and the frame rate.

[0018] As an optional embodiment of the present disclosure, inputting the preset Gaussian noise, the target video prompt word and the first image set into a pre-trained video generation model to generate the target video includes:

[0019] Inputting the first image set into a preset adaptation model of the pre-trained video generation model to obtain an output vector of each layer of the network architecture of the preset adaptation model;

[0020] Inputting the preset Gaussian noise and the target video prompt word into the preset Vincent video model of the pre-trained video generation model to obtain an output vector of each layer of the input layer of the preset Vincent video model;

[0021] The output vector of each layer of the network architecture of the preset adaptation model is added to the output vector of each layer of the input layer of the preset Vincent video model to obtain the target video, which is a video based on denoising of the preset Gaussian noise and matches the target video prompt word and the graffiti image in the first image set.

[0022] As an optional embodiment of the present disclosure, the method further includes:

[0023] Acquire a second image set of the training video sample, a video segment, and annotated description text corresponding to each video segment, wherein the second image set includes graffiti images and all-black images in the video frames corresponding to the training video sample;

[0024] In each round of training, the second image set of the training video sample is input into the preset adaptation model, and the output vector of each layer of the network architecture of the preset adaptation model is saved;

[0025] Fixing the network parameters of each layer of the network architecture of the preset Vincent video model;

[0026] Input the annotated description text corresponding to each video clip into the preset Vincent video model to obtain the predicted noise output by the preset Vincent video model; wherein the output vector of each input layer of the preset Vincent video model corresponds to the output vector of each layer of the network architecture of the preset adaptation model superimposed to serve as the input vector of the next layer of the input layer of the preset Vincent video model;

[0027] Back-propagation is performed according to the loss function of the predicted noise calculated by the preset Vincent video model output to adjust the network parameters of each layer of the network architecture of the preset adaptation model to obtain the pre-trained video generation model;

[0028] Among them, the input layer of the preset Vincent video model and the preset adaptation model have the same network architecture.

[0029] As an optional embodiment of the present disclosure, the method further includes:

[0030] For each of the training video samples, the training video sample is cut into multiple video segments according to a preset resolution and a preset video segment duration.

[0031] As an optional embodiment of the present disclosure, the method further includes:

[0032] For each video segment of the training video sample, a preset video language model is used to perform text annotation on any of the video segments to obtain an annotation description text corresponding to each of the video segments.

[0033] According to a second aspect of the embodiments of the present disclosure, a device for generating a video is provided, the device comprising:

[0034] An image acquisition module, configured to acquire a first image set of an input video, wherein the first image set includes scribble images of a first number of video frames of the input video and completely black images of a second number of video frames;

[0035] A video generation module, used for inputting preset Gaussian noise, target video prompt words and the first image set into a pre-trained video generation model to generate a target video;

[0036] The pre-trained video generation model is jointly trained by a preset video model and a preset adaptation model.

[0037] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0038] a memory for storing a computer program product;

[0039] The processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the method described in the first aspect above is implemented.

[0040] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0041] A fifth aspect of an embodiment of the present disclosure provides a computer program product, including computer program instructions, characterized in that when the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0042] According to the technical solution of the embodiments of the present disclosure, by inputting a sparse doodle image of a video (including doodle images of a first number of video frames of the input video and a completely black image of a second number of video frames), in combination with a Gaussian noise image and a target video prompt word, a pre-trained video generation model is used to generate a target video matching the doodle image, etc., thereby realizing a process of automatically converting relatively rough line drawings such as doodle images into animated videos, helping different users to realize their video generation needs, improving the user's interactive experience, reducing the requirements for input information for generating animated videos, and increasing the scope of applicable fields.

[0043] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0045] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0046] Figure 1 A flowchart of a method for video generation according to an embodiment of the method disclosed herein;

[0047] Figure 2 A schematic diagram of a Vincent video model according to an embodiment of the method disclosed herein;

[0048] Figure 3 A flowchart of a method for generating a video according to another embodiment of the method disclosed herein;

[0049] Figure 4A schematic diagram of the training process of a video generation model in a video generation method according to an embodiment of the method disclosed herein;

[0050] Figure 5 A schematic diagram of performing video generation based on a pre-trained video generation model according to another embodiment of the method disclosed herein;

[0051] Figure 6 A structural block diagram of a video generation device according to an embodiment of the present invention;

[0052] Figure 7 A structural block diagram of a video generation device according to another embodiment of the device disclosed herein;

[0053] Figure 8 A structural block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] In recent years, due to the rapid development of video generation model technology, video generation models have made great progress in the field of AIGC (Artificial Intelligence Generated Content). In video generation models, the generation of target videos based on scribbles (graffiti or hand-drawn pictures) has become an important demand in many fields today. For example, in the field of education, some hand-drawn pictures can be used to obtain corresponding target videos through models to teach painting or design skills. For example, in game scenes, the generation of game environment videos based on players' graffiti can enhance the interactive experience.

[0055] In order to enable those skilled in the art to clearly and accurately understand the technical solutions of the embodiments of the present disclosure, the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0056] Figure 1 FIG. 1 is a flow chart of a method for video generation according to an embodiment of the method disclosed herein. Figure 1 As shown, a method for generating a video according to an embodiment of the present disclosure may include the following steps:

[0057] Step 101: Acquire a first image set of an input video, wherein the first image set includes scribble images of a first number of video frames of the input video and completely black images of a second number of video frames.

[0058] The input video can be a known video clip input by the user or generated by AI (Artificial Intelligence), which is the image source of the doodle image.

[0059] In an embodiment of the present disclosure, multiple video frames of an input video can be obtained according to a preset processing method, and a graffiti image of the video frame can be extracted from each video frame to obtain a graffiti image set of the input video, and then the target graffiti image in the graffiti image set can be replaced with a completely black image to obtain a first image set.

[0060] Exemplarily, the input video is processed by a frame extraction tool to obtain multiple video frames, for example, it can be implemented by using a multimedia processing tool such as FFmpeg (Fast Forward Mpeg), and this is not limited in the present disclosure. The video frames of the input video are extracted by using the multimedia processing tool, and the video is converted into multiple images, which lays a data foundation for the extraction of the scribble image, reduces the consumption of video processing resources, and improves the efficiency of the scribble image extraction.

[0061] The graffiti image set includes a plurality of graffiti images, and the target graffiti image is randomly extracted from the graffiti image set. Therefore, replacing the target graffiti image in the graffiti image set with a completely black image means randomly extracting a portion of the graffiti images from the graffiti image set, and placing the randomly extracted portion of the graffiti images in the initialized completely black video according to the timing of the corresponding video clips, so that the extracted graffiti images and the video frames of the replaced completely black video finally constitute the first image set. Therefore, the method of randomly extracting graffiti images from the graffiti image set to replace the completely black images makes the formed first image set more diverse, not limited by the timing of the video clips, more versatile and easy to implement.

[0062] As an embodiment, the total number of frames of the video frames of the input video can be determined first, and then based on the total number of frames of the video frames of the input video, a preset proportion of graffiti images in the graffiti image set can be determined as the target graffiti images, so as to replace the target graffiti images with all-black images to obtain a first image set. The total number of frames of the video frames of the input video can be determined by obtaining the total duration and frame rate of the input video, and then calculating the total number of frames of the video frames of the input video according to the total duration and frame rate. In this embodiment, graffiti images are extracted from the graffiti image set according to a preset ratio to replace the all-black images, so that the ratio of graffiti images and all-black images in the generated first image set is fixed, so as to avoid too many all-black images or too many graffiti images, which may affect the effect of the final target video and improve the quality of the target video. In addition, the extraction of graffiti images can be reduced, the use of image processing resources can be reduced, and the image processing efficiency can be improved.

[0063] For example, the video clip can be cropped into multiple video frames according to a preset duration and / or a preset resolution. For example, a video clip with a duration of 10 seconds and a frame rate of 16 can be cropped into 160 frames of images, and graffiti images are extracted from the 160 frames of images to obtain 160 frames of graffiti images. Then, for example, 85% of the graffiti images are determined from the 160 frames of graffiti images, and these graffiti images are determined as target graffiti images. A corresponding number of all-black images are used and replaced accordingly according to the timing of the target graffiti images to obtain a first image set.

[0064] As another embodiment of the present disclosure, the video clip can be cropped into multiple video frames according to a preset duration and / or preset resolution. For example, a video clip with a duration of 10 seconds and a frame rate of 16 can be cropped into an image of 160 frames, and then 15% of the video frames are randomly obtained from the 160 frames, and scribble images are extracted from them. The scribble images of the 15% video frames are all replaced with all-black images according to the time sequence of the video clip, and 85% of the scribble images and 15% of the all-black images are reassembled into a first image set. According to this embodiment, the extraction of scribble images can be reduced, the use of image processing resources can be reduced, and the image processing efficiency can be improved.

[0065] Among them, the graffiti image is a Scribble image, and its extraction can be achieved through the Scribble model. The Scribble model is a preprocessor for generating graffiti or sketch effects. It can convert the input image into a form similar to graffiti or sketch, making the image look more concise and abstract. In addition, the Scribble image can help the video generation model better understand the main features and structures in the image, so as to better process and identify the content in the image, and then guide the video generation model based on the Scribble image to obtain a better target video. There are many applicable scenarios, which is conducive to the universality of AIGC (Artificial Intelligence Generated Content) technology. The Scribble model is a component provided by ControlNet, which more accurately identifies and extracts the contour information of the image in image processing, helping subsequent processing algorithms or machine learning models to better understand and process the edge information in the image.

[0066] Step 102: input preset Gaussian noise, target video prompt words and the first image set into a pre-trained video generation model to generate a target video.

[0067] The pre-trained video generation model is jointly trained by a preset Vincent video model and a preset adaptation model. Optionally, in the process of jointly training the preset Vincent video model and the preset adaptation model to obtain the pre-trained video generation model, first obtain a preset Vincent video model that has video generation capability but cannot generate high-quality videos for inputs including graffiti images (the model has high input requirements), and obtain a preset adaptation model to be trained, wherein the network layer of the preset adaptation model (i.e., each computing layer included in the preset adaptation model) is mapped one-to-one with the network layer of the input layer of the preset Vincent video model, and the corresponding computing layers have the same structure. Then, the second video clip in the training video sample is input into the preset adaptation model, and the output vector of each layer of the preset adaptation model is saved. And, multiple video clips and annotated description texts in the training video sample are input into the preset Vincent video model, and in each time step of the preset Vincent video model denoising process, the output vector of each computing layer of the input layer of the preset Vincent video model is superimposed with the output vector of the computing layer corresponding to the preset adaptation model (for example, vector weighted summation calculation), and the processing result is used as the input vector of the next computing layer in the input layer. At the same time, through the loss function of the preset Vincent video model, the network parameters of each computing layer of the preset adaptation model are adjusted, and then the preset adaptation model is trained, and then the pre-trained video generation model is obtained by jointly training the preset Vincent video model and the preset adaptation model.

[0068] In an embodiment of the present invention, by jointly training a preset adaptation model and a preset Vincent video model to obtain a pre-trained video generation model, it is possible not to directly adjust the parameters of the preset Vincent video model (the magnitude of the parameters of the preset Vincent video model is generally large, such as billions or tens of billions), which can not only reduce the number of parameters required for training, but also reduce the demand for computing resources during training, thereby speeding up the training speed. At the same time, when the preset Vincent video model performs a specific task, the output vector of the input layer of the preset Vincent video model is fine-tuned through the preset adaptation model, thereby achieving the specific task can be completed efficiently and accurately without the need for large-scale overall training of the preset Vincent video model. It can be seen from this that the pre-trained video generation model obtained by jointly training the preset adaptation model and the preset Vincent video model realizes the rapid generation of high-quality target videos based on the AIGC technology through the first video clip including the graffiti image, and does not affect the versatility of the preset Vincent video model.

[0069] In the disclosed embodiment, the target video prompt word serves as input text information to guide the generation of the target video, and is used to describe the content requirements of the target video or the image feature words of the target video. The target video prompt word is used to describe the content of the target video to generate a target video that matches the description of the target video prompt word, so that the video generation model can generate a target video that better meets the user's expectations and improve the user experience.

[0070] The video generation model (i.e., the pre-trained video generation model) provided in the embodiments of the present disclosure is jointly trained by a preset Vincent video model and a preset adaptation model, wherein the preset Vincent video model is a diffusion model structure. The diffusion model is a generation model based on a deep learning algorithm, which generates high-quality content through a gradual denoising and denoising process, performs well in generating images, videos, audio, and other high-dimensional data, and has powerful performance in generation tasks. In the present disclosure, the diffusion model is used as the preset Vincent video model as a pre-trained model for video generation. The diffusion model can generate a latent space representation of the target video based on the latent space representation of the target video prompt word and the first image set, and further map the latent space representation of the target video to the pixel space through a decoder to obtain the target video. Among them, the latent space representations of the target video prompt words and the first image set use a text encoder and an image encoder respectively. For example, the target video prompt words and the first image set are encoded by a text encoder and a variational auto-encoder (VAE) as an image encoder respectively to obtain the corresponding latent space expressions. The text encoder and the image encoder are responsible for mapping the text data and video data from the original pixel space to the latent space (LatentSpace) to obtain an implicit and continuous representation, which can make the model calculation more efficient; and the decoder can reconstruct the original video from the latent space representation and map the generation result of the diffusion model from the latent space to the pixel space.

[0071] As an embodiment of the present disclosure, the preset text-generated video model may be a Stable diffusion model. Stable diffusion is a text-image generation model based on a latent diffusion model, which generates images by iteratively "denoising" data in a latent representation space, and then decodes the representation result into a complete image, so that the text-image generation can be used on consumer-grade GPUs.

[0072] Figure 2 FIG. 1 is a schematic diagram of the structure of a Vincent video model according to an embodiment of the method disclosed herein. Figure 2As shown, in the embodiment of the present disclosure, Stable diffusion (hereinafter referred to as diffusion model) is used as the video model, and the diffusion model includes a diffusion process (also called a forward process or a noise adding process) and a generation process (also called a backward process or a denoising process). The diffusion process is a predefined determination process, and the samples of the Gaussian noise distribution finally generated by it can be used as supervision samples of the generation process.

[0073] Specifically, the diffusion process provides labeled samples for model training. In this process, known Gaussian noise is gradually added to the sample image. After each addition of Gaussian noise, a sample image with Gaussian noise is output, for example Figure 2 In the diffusion process direction x0 is the sample image (a clear image), x1, x2, x3, etc. are all sample images with Gaussian noise, until x0 completely becomes the standard Gaussian noise Z.

[0074] The generation process starts with standard Gaussian noise, for example, starting from Z, and gradually removes Gaussian noise to obtain a picture with less Gaussian noise, such as x3, x2, x1... in the figure, the Gaussian noise gradually decreases until a clear picture x0 is generated. In this process, how many steps of denoising are needed and how much noise is removed in each step are guided by the process of gradually adding Gaussian noise in the diffusion process to guide the denoising process in the generation stage, which can be used as supervised learning in the generation stage.

[0075] The preset adaptation model may be, for example, an Adapter model. In the disclosed embodiment, the purpose of the preset adaptation model (eg, Adapter) is to generate a target video based on various image features of an input sparse graffiti image, and the generated target video is a video that matches the input sparse graffiti image.

[0076] In the disclosed embodiment, the structure of the preset adaptation model is the same as the input layer structure of the preset Vincent video model. The first image set can be first input into the preset adaptation model of the pre-trained video generation model to obtain the output vector of each layer of the network architecture of the preset adaptation model. The preset Gaussian noise and the target video prompt word are input into the preset Vincent video model of the pre-trained video generation model to obtain the output vector of each layer of the input layer of the preset Vincent video model. The output vector of each layer of the network architecture of the preset adaptation model is added to the output vector of each layer of the input layer of the preset Vincent video model to obtain the target video. The target video is a video based on denoising of the preset Gaussian noise and matching the target video prompt word and the sparse graffiti image in the first image set.

[0077] Specifically, at each time step in the denoising process of the preset Vincent video model, the output vector of each layer of the preset adaptation model is added to the output vector of each layer of the input layer of the preset Vincent video model to obtain a calculation result, and the calculation result is used as the input of the next layer in the input layer of the preset Vincent video model. The preset Vincent video model is subjected to T rounds of iteration (i.e., after T time steps) of denoising to obtain the final target video. Among them, the preset Vincent video model includes an input layer and an output layer, and the input layer of the preset Vincent video model includes multiple calculation layers. The multiple calculation layers of the input layer of the preset Vincent video model are mapped one-to-one with the multiple calculation layers of the preset adaptation model, and the structure of each group of corresponding calculation layers is the same. That is, after the output layer of the preset Vincent video model outputs the output vector of the first round, it is input into the input layer of the preset Vincent video model again as the initial input vector of the second round. After the denoising is completed after T rounds of loop execution according to the above process, the final output vector of the output layer of the preset Vincent video model is decoded by the decoder, and it is mapped from the latent space to the pixel space to obtain the target video.

[0078] In this embodiment, since the structure of the preset adaptation model is the same as the structure of the input layer of the preset Vincent video model, feature vectors of the same dimension can be extracted for the input information of each model, and then a second video clip that highly matches the conditional information generated by the guide video can be obtained through step-by-step denoising.

[0079] exist Figure 1 Based on the embodiment shown, the embodiment of the present disclosure can also be implemented as follows Figure 3 The embodiment shown. Figure 3 FIG. 1 is a flow chart of a method for generating a video according to another embodiment of the method disclosed herein. Figure 3 As shown, a method for generating a video according to an embodiment of the present disclosure may include the following steps:

[0080] Step 301, jointly train a preset raw video model and a preset adaptation model to obtain a pre-trained video generation model.

[0081] As mentioned above, in the embodiment of the present disclosure, Stable diffusion (hereinafter referred to as diffusion model) can be used as the Vincent video model, and Adapter can be used as the preset adaptation model. In the joint training process, the preset adaptation model Adapter inserts the output vector of each layer in each layer of the pre-trained video generation model into the parameters used for downstream tasks (i.e., the training of the preset Vincent video model), and uses the preset Vincent video model as the model body, and its network parameters are frozen, that is, only the parameters of a specific task (such as the generation task of the target video affected by the graffiti image in the embodiment of the present disclosure) are trained, which can reduce the computing power overhead during training.

[0082] For example, it can be achieved by Figure 4 The flowchart shown implements pre-training of the video generation model. Figure 4 FIG. 1 is a schematic diagram of a training process of a video generation model in a video generation method according to an embodiment of the present disclosure. Figure 4 As shown, the pre-training process of the video generation model in step 301 can be implemented by the following steps:

[0083] Step 3011, obtaining a second image set of training video samples, video clips, and annotated description text corresponding to each video clip.

[0084] The second image set, the video clips, and the annotated description text corresponding to each video clip are all training data for the video generation model.

[0085] The second image set includes graffiti images and all-black images in the video frames of the corresponding training video samples. Exemplarily, a large number of complete videos can be collected as training video samples, and data preprocessing can be performed on these training video samples. For example, these videos are cut into video clips with fixed resolution and fixed length, and then the graffiti images of these complete videos are extracted through, for example, the Scribble model, and then the randomly determined target graffiti images in the extracted graffiti images are replaced with all-black images. For example, a 10-second video with a frame rate of 16 has a total of 160 frames. The graffiti images of the 160 frames are extracted, and 15% of the graffiti images are extracted and replaced with all-black images in the time sequence of these graffiti images. The replaced image set includes 85% of the graffiti images and 15% of the all-black images, which constitute the second image set.

[0086] As an embodiment of the present disclosure, in the process of obtaining multiple video clips of each training video sample, the training video sample can be cut into multiple video clips according to a preset resolution and a preset video segment length. Furthermore, for each video clip, a preset video language model can be used to perform text annotation on any video clip to obtain an annotation description text corresponding to each video clip. For example, a 10-second video with a frame rate of 16, for example, is cut into video clips (or images) of 160 video frames according to a fixed length, and then for the video clips of 160 video frames, a preset video language model is used to perform text annotation on the video clips of each video frame to obtain an annotation description text for each video clip. The annotation description text completes the description of the video clip of any video frame through text or words, and is finally used for model training, so that the model obtains more image features and information. The complete training video sample is cropped into multiple video clips. The objects or states contained in each video clip may be different. In this way, text annotation can be performed in a more fine-grained manner, and the objects or states in the video clips can be described more clearly with text, making the annotation description text more specific and accurate, providing more fine-grained training data for model training and improving the robustness of the model.

[0087] Among them, the preset video language model is, for example, the CogVideoX series model. Those skilled in the art will appreciate that the embodiments of the present disclosure do not limit the type or category of the preset video language model, as long as it can implement text annotation of video clips.

[0088] Step 3012: input the second image set of the training video samples into the preset adaptation model, and save the output vector of each layer of the network architecture of the preset adaptation model.

[0089] Input multiple second image sets corresponding to multiple training video samples into the preset adaptation model, obtain the output vector of each layer of the network architecture of the preset adaptation model and save it. The output vector is the parameter vector of each layer of the preset adaptation model obtained after continuous adjustment by the loss function (input by the preset Wensheng video model in the present embodiment) during training, that is, the feature vector of each frame of the second image set extracted by the preset adaptation model during the training process. The trained preset adaptation model can extract the feature vector corresponding to the image when receiving an image such as the second image set.

[0090] Step 3013, fix the network parameters of each layer of the network architecture of the preset Vincent video model.

[0091] In each round of training, for example, the network parameters of each layer of the network architecture of the preset Vincent video model can be set to fixed values ​​(such as False), so that the gradients of these parameters will no longer be calculated during the back propagation process, and thus will not be updated, thereby achieving the purpose of fixing the network parameters of the preset Vincent video model. In the disclosed embodiment, the preset Vincent video model is used as one of the starting models of the pre-trained video generation model. By fixing the network parameters of the preset Vincent video model, the risk of overfitting can be reduced, the amount of calculation can be reduced, and the memory consumption can be reduced.

[0092] Step 3014, in each round of training, the annotated description text corresponding to each video clip is input into the preset Vincent video model to obtain the predicted noise output by the preset Vincent video model; wherein the output vector of each input layer of the preset Vincent video model corresponds to the output vector of each layer of the network architecture of the superimposed preset adaptation model, so as to serve as the input vector of the next layer of the input layer of the preset Vincent video model.

[0093] In the embodiments of the present disclosure, the network structures of the preset Vincent video model and the preset adaptation model can be preset. For example, the network architecture of the preset adaptation model is set to be the same as the network architecture of the input layer of the preset Vincent video model. In this way, the output vector of each layer of the preset adaptation model and the output vector of each layer in the input layer of the forward propagation stage of the preset Vincent video model have the same data dimension.

[0094] In each round of training, the annotated description text corresponding to each video clip is used as the input of the forward propagation of the preset Vincent video model to obtain the output vector of each layer of the input layer of the forward propagation stage of the preset Vincent video model. Then, the output vector of each layer of the preset adaptation model saved in step 3012 is sequentially superimposed on the output vector of each layer of the input layer of the forward propagation stage of the preset Vincent video model according to the hierarchy of the network architecture, and after superposition, it is used as the input vector of the input layer of the next forward propagation stage, and finally the predicted noise output by the preset Vincent video model is obtained.

[0095] The forward propagation of the preset Vincent video model gradually adds noise to each video clip, and the noise is Gaussian distribution matrix noise. After each forward propagation of the preset Vincent video model, the noise of the video clip being processed will increase compared with the previous network. Assuming that x1 Gaussian noise is added to each layer, the total forward propagation stage includes 10 steps. For the original input video clip x0, during the forward propagation stage of the preset Vincent video model, 10 times x1 Gaussian noise will be added to the image of x0. In the forward propagation of the preset Vincent video model, the goal is to process the original input video clip into a standard Gaussian noise by gradually adding Gaussian noise, so the amount of Gaussian noise added in each step can be fixed.

[0096] Therefore, in the forward propagation of the preset Vincent video model, the output of each layer in its input layer is also a video clip with Gaussian noise, that is, the predicted noise output by the preset Vincent video model, which is a vector obtained by adding the feature vector of the video clip to the Gaussian noise matrix.

[0097] Step 3015, back propagation is performed based on the loss function of the predicted noise calculated according to the preset Vincent video model output to adjust the network parameters of each layer of the network architecture of the preset adaptation model to obtain a pre-trained video generation model.

[0098] In the disclosed embodiment, in each round of training, the preset Vincent video model can predict the noise to calculate the loss function, and back-propagate the loss function. In the back-propagation process, since the network parameters are fixed, the network parameters of the preset adaptation model will not be adjusted, but the network parameters of each layer of the network architecture of the preset adaptation model can be adjusted to achieve optimized training of the preset adaptation model, thereby improving the performance of the preset adaptation model and enhancing the robustness and scalability of the video generation model. In this way, the preset Vincent video model and the preset adaptation model are jointly trained to obtain the adaptation generation model.

[0099] It should be noted that the disclosed embodiment is to realize the generation of the target video affected by the graffiti image, and the preset Vincent video model is used as the main model. During the training process, the parameters of the preset Vincent video model can be frozen, and only the preset adaptation model can be trained, so that the network layer parameters of the preset Vincent video model do not change with the training process. The embodiment of the present invention not only does not change the universality of the preset Vincent video model, but also the preset adaptation model structure and parameters are lightweight, so the generation of a complete video based on the graffiti image is realized quickly and at low cost.

[0100] Step 302: Acquire a first image set of the input video, wherein the first image set includes scribble images of a first number of video frames of the input video and completely black images of a second number of video frames.

[0101] This step can refer to Figure 1 For the sake of brevity, step 101 is not described again here.

[0102] Step 303: input the preset Gaussian noise, the target video prompt word and the first image set into a pre-trained video generation model to generate the target video.

[0103] This step can refer to Figure 1 For the sake of brevity, step 102 is not described again here.

[0104] In summary, according to the technical solution of the embodiment of the present disclosure, by inputting a sparse graffiti image of a video (including graffiti images of a first number of video frames of the input video and a completely black image of a second number of video frames), combined with a Gaussian noise image and a target video prompt word, a pre-trained video generation model is used to generate a target video matching the graffiti image, etc., so as to realize the process of automatically converting relatively rough line drawings such as graffiti images into animated videos, thereby helping different users to realize their video generation needs, improving the user's interactive experience, reducing the requirements for input information for generating animated videos, and increasing the scope of applicable fields.

[0105] For example, see Figure 5 As shown, a schematic diagram of video generation based on a pre-trained video generation model in another embodiment of the method disclosed herein. Figure 5As shown, the preset adaptation model takes the first image set (a combination of graffiti images and all-black images) as input, firstly extracts image features from the images in the first image set through its encoder (variational autoencoder), and then, for example, by processing the image features, outputs a vector matrix corresponding to the image features (i.e., the output vector of each layer of the network architecture of the preset adaptation model), that is, firstly extracts the images in the first image set through its encoder (e.g., VAE variational autoencoder) to obtain the latent space expression of the first image set, and the latent space expression of the first image set is input into the preset adaptation model in the pre-trained video generation model to obtain the output vector of each computing layer in the preset adaptation model, and the output vector of each computing layer is saved. The preset Gaussian noise and the target video prompt word are passed through the encoder to obtain the corresponding expression vector, wherein the target video prompt word is text information, and the corresponding text expression vector can be obtained by, for example, text encoding processing of TextEncoder. The preset Gaussian noise and the expression vector corresponding to the target video prompt word are input into the preset Vincent video model. Since the multiple computing layers of the input layer of the preset Vincent video model and the multiple computing layers of the preset adaptation model are mapped one by one, the preset Vincent video model obtains the output vector of each computing layer in the preset adaptation model at each time step of the denoising process, and adds the output vector of each computing layer of the input layer of the preset Vincent video model to the output vector of the corresponding computing layer of the preset adaptation model, and the obtained calculation result is used as the input vector of the next computing layer in the input layer of the preset Vincent video model. And, the output vector of the output layer of the preset Vincent video model can be used as the input vector of the input layer of the preset Vincent video model at the next time step. It can be understood that after the denoising process of T time steps, the latent space expression of the target video is obtained, and the latent space expression of the target video can be obtained by the decoder to obtain the target video in the pixel space.

[0106] Correspondingly, the embodiments of the present disclosure also provide device embodiments corresponding to the aforementioned method embodiments. Figure 6 This is a structural block diagram of a video generation device according to an embodiment of the present disclosure. Figure 6 As shown, a video generation device according to an embodiment of the present disclosure may include an image acquisition module 601 and a video generation module 602, wherein:

[0107] An image acquisition module 601 is used to acquire a first image set of an input video, wherein the first image set includes scribble images of a first number of video frames of the input video and completely black images of a second number of video frames;

[0108] The video generation module 602 is used to input preset Gaussian noise, target video prompt words and the first image set into a pre-trained video generation model to generate a target video;

[0109] The pre-trained video generation model is jointly trained by a preset video model and a preset adaptation model.

[0110] In summary, according to the technical solution of the embodiment of the present disclosure, by inputting a sparse graffiti image of a video (including graffiti images of a first number of video frames of the input video and a completely black image of a second number of video frames), combined with a Gaussian noise image and a target video prompt word, a pre-trained video generation model is used to generate a target video matching the graffiti image, etc., so as to realize the process of automatically converting relatively rough line drawings such as graffiti images into animated videos, thereby helping different users to realize their video generation needs, improving the user's interactive experience, reducing the requirements for input information for generating animated videos, and increasing the scope of applicable fields.

[0111] exist Figure 6 Based on the device embodiment shown, the present disclosure also provides another device embodiment. Figure 7 This is a structural block diagram of a video generation device according to another embodiment of the present invention. Figure 7 As shown, a video generation device according to an embodiment of the present disclosure may include an image acquisition module 601, a video generation module 602, a training data acquisition module 603, a first training module 604, a second training module 605, a video generation model determination module 606, a video cropping module 607, and a video annotation module 608. Among them:

[0112] The training data acquisition module 603 is used to acquire a second image set of the training video sample, a video segment, and a label description text corresponding to each video segment, wherein the second image set includes graffiti images and all-black images in the video frame corresponding to the training video sample;

[0113] A first training module 604 is used to input the second image set of the training video sample into the preset adaptation model in each round of training, and save the output vector of each layer of the network architecture of the preset adaptation model;

[0114] The second training module 605 is used to fix the network parameters of each layer of the network architecture of the preset Vincent video model, and in each round of training, input the annotation description text corresponding to each video clip into the preset Vincent video model to obtain the predicted noise output by the preset Vincent video model; wherein the output vector of each input layer of the preset Vincent video model corresponds to the output vector of each layer of the network architecture of the preset adaptation model superimposed to serve as the input vector of the next layer of the input layer of the preset Vincent video model;

[0115] The video generation model determination module 606 is used to perform back propagation according to the loss function of the predicted noise calculated by the preset Vincent video model output to adjust the network parameters of each layer of the network architecture of the preset adaptation model to obtain the pre-trained video generation model; wherein the input layer of the preset Vincent video model and the preset adaptation model have the same network architecture;

[0116] The video cropping module 607 is used to crop each of the training video samples into multiple video segments according to a preset resolution and a preset video segment duration;

[0117] The video annotation module 608 performs text annotation on each video segment of the training video sample using a preset video language model to obtain an annotation description text corresponding to each video segment;

[0118] An image acquisition module 601 is used to acquire a first image set of an input video, wherein the first image set includes scribble images of a first number of video frames of the input video and completely black images of a second number of video frames;

[0119] The video generation module 602 is used to input preset Gaussian noise, target video prompt words and the first image set into a pre-trained video generation model to generate a target video.

[0120] As an optional embodiment of the present disclosure, the image acquisition module 601 may further include:

[0121] A first acquisition unit, configured to acquire a plurality of video frames of the input video according to a preset processing method;

[0122] A scribble image extraction unit, used to extract the scribble image of the video frame from each of the video frames to obtain a scribble image set of the input video;

[0123] The image replacement unit is used to replace the target graffiti image in the graffiti image set with a completely black image to obtain the first image set.

[0124] Wherein, the graph replacement unit may further include:

[0125] A total number of video frames determining subunit, used to determine the total number of video frames of the input video;

[0126] The graffiti image replacement subunit is used to determine a preset proportion of graffiti images in the graffiti image set as target graffiti images based on the total number of video frames of the input video, so as to replace the target graffiti images with all-black images to obtain the first image set.

[0127] Furthermore, the total number of video frames determining subunit may obtain the total duration and frame rate of the input video, and then calculate the total number of video frames of the input video according to the total duration and frame rate.

[0128] As an optional embodiment of the present disclosure, the video generation module 602 may further include:

[0129] A first vector output unit, used to input the first image set into a preset adaptation model of the pre-trained video generation model to obtain an output vector of each layer of the network architecture of the preset adaptation model;

[0130] A second vector output unit, used for inputting the preset Gaussian noise and the target video prompt word into a preset Vincent video model of the pre-trained video generation model to obtain an output vector of each layer of the input layer of the preset Vincent video model;

[0131] A vector addition unit is used to add the output vector of each layer of the network architecture of the preset adaptation model and the output vector of each layer of the input layer of the preset Vincent video model to obtain the target video, wherein the target video is a video that matches the target video prompt word and the graffiti image in the first image set based on the preset Gaussian noise denoising process.

[0132] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present disclosure. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0133] Below, reference Figure 8 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0134] Figure 8A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0135] like Figure 8 As shown, the electronic device includes one or more processors and memory.

[0136] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0137] The memory may store one or more computer program products, and the memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program products may be stored on the computer-readable storage medium, and the processor may run the computer program product to implement the video generation method of each embodiment of the present disclosure described above and / or other desired functions.

[0138] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0139] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0140] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0141] Of course, to simplify, Figure 8 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0142] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the method of video generation according to various embodiments of the present disclosure described in the above part of this specification.

[0143] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0144] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps in the method of video generation according to various embodiments of the present disclosure described in the above part of this specification.

[0145] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0146] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0147] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0148] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0149] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0150] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0151] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0152] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for video generation, characterized in that: The method comprises: Acquire a first image set of an input video, wherein the first image set includes scribble images of a first number of video frames of the input video and completely black images of a second number of video frames; Inputting preset Gaussian noise, target video prompt words and the first image set into a pre-trained video generation model to generate a target video, including: inputting the first image set into a preset adaptation model of the pre-trained video generation model to obtain an output vector of each layer of the network architecture of the preset adaptation model; inputting the preset Gaussian noise and the target video prompt words into a preset Vincent video model of the pre-trained video generation model to obtain an output vector of each layer of the input layer of the preset Vincent video model; adding the output vector of each layer of the network architecture of the preset adaptation model to the output vector of each layer of the input layer of the preset Vincent video model to obtain the target video, wherein the target video is a video based on denoising of the preset Gaussian noise and matching the target video prompt words and the graffiti in the first image set; The pre-trained video generation model is jointly trained by the preset video model and the preset adaptation model.

2. The method according to claim 1, characterized in that The step of obtaining a first image set of an input video comprises: Acquire multiple video frames of the input video according to a preset processing method; Extracting a graffiti image of the video frame from each of the video frames to obtain a graffiti image set of the input video; The target graffiti image in the graffiti image set is replaced with a completely black image to obtain the first image set.

3. The method according to claim 2, characterized in that The step of replacing the target graffiti image in the graffiti image set with a completely black image to obtain the first image set comprises: Determine the total number of video frames of the input video; Based on the total number of video frames of the input video, determining a preset proportion of graffiti images in the graffiti image set as target graffiti images; The target graffiti image is replaced with a completely black image to obtain the first image set.

4. The method according to claim 3, characterized in that The determining the total number of video frames of the input video comprises: Obtain the total duration and frame rate of the input video; The total number of video frames of the input video is calculated according to the total duration and the frame rate.

5. The method according to claim 1, characterized in that The method further comprises: Acquire a second image set of the training video sample, a video segment, and annotated description text corresponding to each video segment, wherein the second image set includes graffiti images and all-black images in the video frames corresponding to the training video sample; Inputting the second image set of the training video sample into the preset adaptation model, and saving the output vector of each layer of the network architecture of the preset adaptation model; Fixing the network parameters of each layer of the network architecture of the preset Vincent video model; In each round of training, the annotated description text corresponding to each video clip is input into the preset Vincent video model to obtain the predicted noise output by the preset Vincent video model; wherein the output vector of each input layer of the preset Vincent video model corresponds to the output vector of each layer of the network architecture of the preset adaptation model superimposed to serve as the input vector of the next layer of the input layer of the preset Vincent video model; Back-propagation is performed according to the loss function of the predicted noise calculated by the preset Vincent video model output to adjust the network parameters of each layer of the network architecture of the preset adaptation model to obtain the pre-trained video generation model; Among them, the input layer of the preset Vincent video model and the preset adaptation model have the same network architecture.

6. The method according to claim 5, characterized in that The method further comprises: For each of the training video samples, the training video sample is cut into multiple video segments according to a preset resolution and a preset video segment duration.

7. A video generation device, characterized in that: The device comprises: An image acquisition module, configured to acquire a first image set of an input video, wherein the first image set includes scribble images of a first number of video frames of the input video and completely black images of a second number of video frames; A video generation module, used for inputting preset Gaussian noise, target video prompt words and the first image set into a pre-trained video generation model to generate a target video, including: inputting the first image set into a preset adaptation model of the pre-trained video generation model to obtain an output vector of each layer of the network architecture of the preset adaptation model; inputting the preset Gaussian noise and the target video prompt words into a preset Vincent video model of the pre-trained video generation model to obtain an output vector of each layer of the input layer of the preset Vincent video model; adding the output vector of each layer of the network architecture of the preset adaptation model to the output vector of each layer of the input layer of the preset Vincent video model to obtain the target video, wherein the target video is a video based on denoising of the preset Gaussian noise and matching the target video prompt words and the graffiti in the first image set; The pre-trained video generation model is jointly trained by the preset video model and the preset adaptation model.

8. An electronic device, characterized in that: include: A memory for storing a computer program product; A processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, it implements the method described in any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method described in any one of claims 1 to 6 is implemented.

Images