Video generation method and device, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention discloses a video generation method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: obtaining graffiti maps of all video frames of an input video, combining Gaussian noise and condition information for guiding the generation of the video, and generating a target video through a pre-trained video generation model. The process of automatically converting rough line diagrams such as graffiti diagrams into animation videos is achieved, different users are helped to meet the video generation requirements, the interactive experience of the users is improved, the requirement for input information generated by the animation videos is lowered, the range of the application field is widened, and in addition, the graffiti diagrams of all video frames of the input videos are utilized, so that the user experience is improved. More image feature information can be provided for the pre-trained video generation model, the matching degree of the target video and the graffiti map is improved, and the generation quality of the target video is improved.

Description

Technical Field

[0001] The present disclosure relates to video generation technology, and in particular to a video generation method, device, electronic device, storage medium and program product. Background Art

[0002] Due to the rapid development of video generation model technology, video generation models have made great progress in the field of AIGC (Artificial Intelligence Generated Content). With the continuous advancement of technology, the application prospects of image-generated video technology in the field of AIGC technology are very broad, and may play an important role in many fields such as film and television production, advertising marketing, education, and entertainment games, and promote the innovation and development of related industries.

[0003] In today's education field, AICG and other image-generated video models are urgently needed to make teaching more vivid and vivid. For example, some simple hand-drawn pictures (or doodles, etc.) can be used to generate more vivid videos through models to teach painting or design skills, and students can see their own doodles transformed into animations, which helps students cultivate creativity and stimulate imagination. Summary of the invention

[0004] In order to solve the technical problems in the related art, the embodiments of the present disclosure provide a method, an apparatus, an electronic device, a storage medium and a program product for generating a video.

[0005] A first aspect of the present disclosure provides a method for generating a video, the method comprising: Obtaining a first video segment of an input video, wherein the first video segment includes doodle images of all video frames of the input video; Inputting preset Gaussian noise and condition information guiding video generation into a pre-trained video generation model to generate a target video, wherein the condition information at least includes the first video clip; The pre-trained video generation model is jointly trained by a preset video model and a preset adaptation model.

[0006] As an optional embodiment of the present disclosure, the obtaining a first video segment of the input video includes: Acquire multiple video frames of the input video according to a preset processing method; The scribble image is extracted from each of the video frames to obtain the first video segment, wherein the timing of the scribble image in the first video segment is the same as the timing of the video frames in the input video.

[0007] As an optional embodiment of the present disclosure, the step of inputting the preset Gaussian noise and the condition information guiding the video generation into the pre-trained video generation model to generate the target video includes: Inputting the first video clip into a preset adaptation model of the pre-trained video generation model to obtain an output vector of each layer of the network architecture of the preset adaptation model; Based on the preset Gaussian noise, at each time step, the output vector of each layer of the input layer of the preset video model and the output vector of each layer of the preset adaptation model are weightedly calculated to obtain the target video.

[0008] As an optional embodiment of the present disclosure, the method further includes: Acquire multiple second video clips corresponding to multiple training video samples, multiple video clips of each of the training video samples, and annotated description text corresponding to each of the video clips, wherein the second video clips include doodles of all video frames of the training video samples; Inputting the plurality of second video clips into the preset adaptation model, and saving an output vector of each layer of the preset adaptation model; In each round of training, the annotated description text corresponding to each video segment in the multiple video segments of each training video sample is input into the preset video model to obtain the predicted noise of the preset video model, and the output vector of the preset video model is correspondingly superimposed on the output vector of the input layer of each layer of the preset video model as the input vector of the next layer; adjusting the network parameters of the preset adaptation model based on the predicted noise of the preset video model to obtain the video generation model, Among them, the input layer of the preset video model and the preset adaptation model have the same network architecture.

[0009] As an optional embodiment of the present disclosure, the adjusting the network parameters of the preset adaptation model based on the predicted noise of the preset video model to obtain the video generation model includes: Fixing the network parameters of each layer of the preset video model; Back propagation is performed according to the loss function of the predicted noise calculation, and the network parameters of each layer of the preset adaptation model are adjusted to obtain the video generation model.

[0010] As an optional embodiment of the present disclosure, the method further includes: Each of the training video samples is cropped according to a preset resolution and a preset video segment duration to obtain a corresponding plurality of video segments.

[0011] According to a second aspect of the embodiments of the present disclosure, a device for generating a video is provided, the device comprising: A scribble image acquisition module, used to acquire a first video segment of an input video, wherein the first video segment includes scribble images of all video frames of the input video; A video generation module, configured to input preset Gaussian noise and condition information guiding video generation into a pre-trained video generation model to generate a target video, wherein the condition information at least includes the first video clip; The pre-trained video generation model is jointly trained by a preset video model and a preset adaptation model.

[0012] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including: A memory for storing a computer program product; The processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the method described in the first aspect above is implemented.

[0013] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0014] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, including computer program instructions, which, when executed by a processor, implement the method described in the first aspect.

[0015] According to the technical solution of the embodiment of the present disclosure, by obtaining graffiti images of all video frames of the input video, and combining them with Gaussian noise and conditional information to guide video generation, a pre-trained video generation model is used to generate a target video, thereby realizing the process of automatically converting relatively rough line drawings such as graffiti images into animated videos, helping different users to achieve their video generation needs, improving the user's interactive experience, reducing the requirements for input information for animated video generation, and increasing the scope of applicable fields. In addition, by using graffiti images of all video frames of the input video, more image feature information can be provided to the pre-trained video generation model, thereby improving the matching degree between the target video and the graffiti images, and improving the generation quality of the target video.

[0016] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0018] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which: Figure 1 A flowchart of a method for video generation according to an embodiment of the method disclosed herein; Figure 2 A schematic diagram of a preset video model according to an embodiment of the method disclosed herein; Figure 3 A flowchart of a method for generating a video according to another embodiment of the method disclosed herein; Figure 4 A schematic diagram of performing video generation based on a pre-trained video generation model according to another embodiment of the method disclosed herein; Figure 5 A structural block diagram of a video generation device according to an embodiment of the present invention; Figure 6 A structural block diagram of a video generation device according to another embodiment of the device disclosed herein; Figure 7 A structural block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In recent years, due to the rapid development of video generation model technology, video generation models have made great progress in the field of AIGC (Artificial Intelligence Generated Content). In video generation models, the generation of target videos based on scribbles (graffiti or hand-drawn pictures) has become an important demand in many fields today. For example, in the field of education, some hand-drawn pictures can be used to generate corresponding target videos through models to teach painting or design skills. For another example, in game scenes, the generation of game environment videos based on players' graffiti can enhance the interactive experience.

[0020] In order to enable those skilled in the art to clearly and accurately understand the technical solutions of the embodiments of the present disclosure, the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0021] Figure 1 FIG. 1 is a flow chart of a method for video generation according to an embodiment of the method disclosed herein. Figure 1 As shown, a method for generating a video according to an embodiment of the present disclosure may include the following steps: Step 101: Obtain a first video segment of an input video, wherein the first video segment includes doodle images of all video frames of the input video.

[0022] The input video may be a known video clip input by a user or generated by AI (Artificial Intelligence). In the embodiment of the present disclosure, the graffiti image is extracted from the input video by using the input video as an image source.

[0023] It is understandable that an input video may include multiple video frames, and corresponding graffiti images can be extracted for each video frame, thereby obtaining multiple graffiti images. In the embodiment of the present disclosure, multiple video frames of the input video can be obtained according to a preset processing method, and then the graffiti images can be extracted from each video frame to obtain the first video segment, and the timing in the obtained first video segment is the same as the timing of the video frame in the input video. Among them, the preset processing method can be, for example, some multimedia processing tools (such as FFmpeg multimedia processing tools, VLC multimedia players with frame extraction functions, VirtualDub, etc.), and the embodiment of the present disclosure does not limit this, as long as video frame extraction can be achieved. Processing the input video into multiple video frames and converting the video into a single frame image can extract graffiti images for each frame image in a finer granularity, obtain more graffiti image data, and provide more image features for the subsequent generation of target videos.

[0024] For example, the video clip can be cropped into multiple video frames according to a preset duration and / or preset resolution. For example, a video clip with a duration of 10 seconds and a frame rate of 16 can be cropped into 160 frames of images, and graffiti images are extracted from the 160 frames of images to obtain 160 frames of graffiti images, which are the first video clip.

[0025] Among them, the scribble image is, for example, a Scribble image, and its extraction can be achieved by the Scribble model. The Scribble model is a preprocessor for generating graffiti or sketch effects, which can convert the input image into a form similar to graffiti or sketch, making the image look more concise and abstract. The Scribble model is a component provided by ControlNet, which more accurately identifies and extracts the contour information of the image in image processing, helping subsequent processing algorithms or machine learning models to better understand and process the edge information in the image.

[0026] The Scribble graph includes the main features and structures of each video frame of the input video, which can not only help the subsequent video generation model better identify and understand the main objects in the input video, ensuring the consistency of the main objects in different video frames in the generated target video; but also the Scribble graph facilitates the full use of the creativity of AI, so that the generated content is not limited to the content of the input video, suggesting the richness of the target video content.

[0027] Therefore, based on the first video clip containing the Scribble graph, the target video obtained by the video generation model can be guided to have better effect, and this has many applicable scenarios, which is conducive to the universality of AIGC (Artificial Intelligence Generated Content) technology.

[0028] Step 102: input preset Gaussian noise and condition information guiding video generation into a pre-trained video generation model to generate a target video, wherein the condition information at least includes a first video clip.

[0029] In the disclosed embodiment, the pre-trained video generation model is jointly trained by a preset video model and a preset adaptation model. Optionally, in the process of jointly training the preset video model and the preset adaptation model to obtain the pre-trained video generation model, first obtain a preset video model that has video generation capability but cannot generate high-quality videos for inputs including graffiti images (the model has high input requirements), and obtain a preset adaptation model to be trained, wherein the network layer of the preset adaptation model (i.e., each computing layer included in the preset adaptation model) is mapped one-to-one with the network layer of the input layer of the preset video model, and the corresponding computing layers have the same structure. Then, the second video clip in the training video sample is input into the preset adaptation model, and the output vector of each layer of the preset adaptation model is saved. And, multiple video clips and annotated description texts in the training video sample are input into the preset video model, and in each time step of the preset video model denoising process, the output vector of each computing layer of the input layer of the preset video model is superimposed with the output vector of the computing layer corresponding to the preset adaptation model (for example, vector weighted summation calculation), and the processing result is used as the input vector of the next computing layer in the input layer. At the same time, through the loss function of the preset video model, the network parameters of each computing layer of the preset adaptation model are adjusted to obtain the preset adaptation model, and then the pre-trained video generation model is obtained by jointly training the preset video model and the preset adaptation model.

[0030] In an embodiment of the present invention, by obtaining a pre-trained video generation model based on joint training of a preset adaptation model and a preset video model, it is not necessary to directly adjust the parameters of the preset video model (the magnitude of the parameters of the preset video model is generally large, such as billions or tens of billions). In this way, not only the number of parameters required for training can be reduced, but also the demand for computing resources during training can be reduced, thereby speeding up the training. At the same time, when the preset video model performs a specific task, the output vector of the input layer of the preset video model is fine-tuned through the preset adaptation model, thereby achieving the specific task that can be completed efficiently and accurately without the need for large-scale overall training of the preset video model. It can be seen from this that the pre-trained video generation model obtained by joint training of the preset adaptation model and the preset video model realizes the rapid generation of high-quality target videos based on the AIGC technology through the first video clip including the graffiti image, and does not affect the versatility of the preset video model.

[0031] Among them, the preset video model can be a diffusion model structure. The diffusion model is a generative model based on a deep learning algorithm, which generates high-quality content through a gradual denoising and denoising process, performs well in generating images, videos, audio and other high-dimensional data, and has strong performance in generation tasks. In the present disclosure, the preset video model is a pre-trained model for video generation, which uses a diffusion model as a model prototype structure, and can generate a latent space representation of a target video based on a latent space representation of conditional information guiding video generation, and further maps the latent space representation of the target video to a pixel space through a decoder to obtain the target video. The latent space representation of the conditional information guiding video generation can be obtained by using a text encoder and an image encoder respectively. For example, a variational auto-encoder (VAE) is used to encode the conditional information guiding video generation to obtain the corresponding latent space expression. The text encoder and the image encoder are responsible for mapping the text data and the video data from the original pixel space to the latent space (Latent Space) respectively to obtain an implicit and continuous representation, which can make the model calculation more efficient; the decoder can reconstruct the original video from the latent space representation and map the generation result of the diffusion model from the latent space to the pixel space.

[0032] As an embodiment of the present disclosure, the preset video model may be a Stable diffusion model. Stable diffusion is a text-image generation model based on a latent diffusion model, which generates images by iteratively "denoising" data in a latent representation space, and then decodes the representation result into a complete image / video, so that the text-image generation can be used on consumer-grade GPUs.

[0033] As an optional embodiment of the present disclosure, the conditional information guiding the generation of the target video may also include a target video prompt word. The target video prompt word is used as input text information to guide the generation of the target video. The content of the target video is described by the target video prompt word to generate a target video that matches the description of the target video prompt word, so that the video generation model can generate a target video that better meets the user's expectations and improve the user experience.

[0034] Figure 2 FIG. 1 is a schematic diagram of a preset video model of an embodiment of the method disclosed herein. Figure 2 As shown, in the embodiment of the present disclosure, Stable diffusion (hereinafter referred to as diffusion model) is used as the preset video model, and the diffusion model includes a diffusion process (also called a forward process or a noise adding process) and a generation process (also called a backward process or a denoising process). The diffusion process is a predefined determination process, and the samples of the Gaussian noise distribution finally generated by it can be used as supervision samples of the generation process.

[0035] Specifically, the diffusion process provides labeled samples for model training. In this process, known Gaussian noise is gradually added to the sample image. After each addition of Gaussian noise, a sample image with Gaussian noise is output, for example Figure 2 In the diffusion process direction x0 is the sample image (a clear image), x1, x2, x3... are all sample images with Gaussian noise, until x0 completely becomes the standard Gaussian noise Z.

[0036] The generation process starts with standard Gaussian noise, for example, starting from Z, and gradually removes Gaussian noise to obtain a picture with less Gaussian noise, such as x3, x2, x1... in the figure, the Gaussian noise gradually decreases until a clear picture x0 is generated. In this process, how many steps of denoising are needed and how much noise is removed in each step are guided by the Gaussian noise gradually added in the diffusion process to guide the denoising process in the generation stage, which can be used as supervised learning in the generation stage.

[0037] The preset adaptation model, such as Adapter, is mainly used to fine-tune a specific task on a pre-trained large model. In the disclosed embodiment, the purpose of the preset adaptation model (such as Adapter) is to generate a target video based on various image features of the input graffiti image, and the visual content in the generated target video matches the visual content corresponding to the graffiti image.

[0038] In the disclosed embodiment, the first video clip can be first input into the preset adaptation model of the pre-trained video generation model to obtain the output vector of each layer of the network architecture of the preset adaptation model, and based on the preset Gaussian noise, at each time step, the output vector of each layer of the input layer of the preset video model and the output vector of each layer of the preset adaptation model are weighted to obtain the target video. Among them, the preset video model includes a downsampling block (i.e., input layer) and an upsampling block (i.e., output layer), and the network structure of the preset adaptation model is exactly the same as the network structure of the downsampling block of the preset video model. In the denoising process of each step of the preset video model, each vector in the vector group output by the preset adaptation model is added to the output vector corresponding to the input layer of the preset video model, and then the output layer of the preset video model is calculated (e.g., weighted calculation) to obtain the final output, i.e., the target video. Specifically, the final output of the preset video model passes through the decoder, maps it from the latent space to the pixel space, and obtains the target video (wherein the duration of the target video is consistent with the duration of the first video clip), that is, after T-step denoising of the preset video model (each denoising step of the preset video model includes the calculation of all input layers and all output layers), the target video is obtained through the decoder.

[0039] exist Figure 1 Based on the embodiment shown, the embodiment of the present disclosure can also be implemented as follows Figure 3 The embodiment shown. Figure 3 FIG. 1 is a flow chart of a method for generating a video according to another embodiment of the method disclosed herein. Figure 3 As shown, a method for generating a video according to an embodiment of the present disclosure may include the following steps: Step 301: Acquire multiple second video segments corresponding to multiple training video samples, multiple video segments of each training video sample, and annotated description text corresponding to each video segment.

[0040] The second video segment includes graffiti images of all video frames of the training video sample.

[0041] Exemplarily, a large number of complete videos can be collected as training video samples, and data preprocessing can be performed on these training video samples. For example, these videos can be cut into video clips with fixed resolution and fixed length, and then scribbles from these complete videos can be extracted through, for example, a scribble model. For example, a 10-second video with a frame rate of 16 has a total of 160 frames. For each of the 160 frames, scribbles can be extracted through, for example, a scribble model, and a total of 160 scribbles are extracted to form a second video clip.

[0042] In addition, the acquisition of multiple video clips of each training video sample can be performed by cutting each training video sample into multiple video clips according to a preset resolution and a preset video segment length. Then, for each video clip, a preset video language model is used to perform text annotation on any video clip to obtain the annotation description text corresponding to each video clip. For example, a 10-second video with a frame rate of 16 is, for example, cut into video clips (or images) of 160 video frames according to a fixed length, and then for the video clips of 160 video frames, a preset video language model is used to perform text annotation on the video clips of each video frame to obtain the annotation description text of each video clip, so as to complete the description of the video clip of any video frame through text or words, such as describing the content and features of the video clip, which is used for model training, so that the model can obtain more image features and information, and the obtained annotation description text will also be more accurate, thereby improving the robustness of the model that needs subsequent training.

[0043] The preset video language model is, for example, a CogVideoX series model. Those skilled in the art will appreciate that the embodiments of the present disclosure do not limit the type or category of the preset video language model, as long as text annotation of video clips can be implemented.

[0044] Step 302: input the plurality of second video clips into a preset adaptation model, and save the output vector of each layer of the preset adaptation model.

[0045] The second video clip is input into the preset adaptation model to obtain the output vector (or parameter vector) of each layer of the network architecture of the preset adaptation model. It can be understood that the output vector of each layer of the network architecture of the preset adaptation model is the feature vector of each frame image in the second video clip extracted by the preset adaptation model during the training process.

[0046] Step 303, in each round of training, the annotated description text corresponding to each video clip in the multiple video clips of each training video sample is input into the preset video model to obtain the predicted noise of the preset video model, and the output vector of the preset video model is superimposed on the output vector of the input layer of each layer of the preset video model as the input vector of the next layer.

[0047] The forward propagation of the preset video model gradually adds noise to each video clip, and processes each video clip into Gaussian noise as the input data of the forward propagation of the preset video model, where the noise is Gaussian distribution matrix noise. After each forward propagation of the preset video model, the noise of the video clip being processed will increase compared with the previous network. Assuming that x1 Gaussian noise is added to each layer, the total forward propagation stage includes 10 steps. Then, for the original input video clip x0, 10 times x1 Gaussian noise will be added to the image of x0 during the forward propagation stage of the preset video model. In the forward propagation of the preset video model, the goal is to process the original input video clip into a standard Gaussian noise by gradually adding Gaussian noise, so the amount of Gaussian noise added in each step can be fixed.

[0048] Therefore, in the forward propagation of the preset video model (for example, the Vincent video model), the output of each layer in its input layer is also a video clip with Gaussian noise, that is, a vector obtained by adding the feature vector of the video clip to the Gaussian noise matrix, and the back propagation of the preset video model is based on the Gaussian distribution matrix noise added in the forward propagation stage of the preset video model as a supervision sample for denoising.

[0049] As mentioned above, in the embodiment of the present disclosure, Stable diffusion (hereinafter referred to as diffusion model) can be used as the preset video model, and Adapter can be used as the preset adaptation model. In the joint training process, the preset adaptation model Adapter inserts the output vector of each layer of the pre-trained video generation model into the parameters used for downstream tasks (i.e., the training of the preset video model), and uses the preset video model as the model body, and its network parameters are frozen, that is, only the parameters of a specific task (such as the generation task of the target video affected by the graffiti image in the embodiment of the present disclosure) are trained, which can reduce the computing power overhead during training.

[0050] Step 304: adjust the network parameters of the preset adaptation model based on the predicted noise of the preset adaptation model to obtain a video generation model.

[0051] In the embodiments of the present disclosure, the network structures of the preset video model and the preset adaptation model can be set in advance. For example, the network architecture of the preset adaptation model is set to be the same as the network architecture of the input layer of the preset video model. In this way, the output vector of each layer of the preset adaptation model and the output vector of each layer in the input layer of the forward propagation stage of the preset video model have the same data dimension.

[0052] In this step, first, the network parameters of each layer of the preset video model are fixed. The embodiment of the present disclosure is to realize the generation of the target video affected by the graffiti image. The preset video model is used as the main model. During the training process, the parameters of the preset video model can be frozen, and only the preset adaptation model is trained, so that the network layer parameters of the preset video model do not change with the training process. Then, the loss function is calculated based on the predicted noise output by the preset video model to perform back propagation, and then the network parameters of each layer of the preset adaptation model are adjusted to finally obtain the video generation model. In this way, the features of the second video clip extracted by the preset adaptation model can be added to the input of the preset video model, which plays a role in prompting the image features of the target video output by the preset video model, and the predicted noise of the preset video model is further used to calculate the loss function to adjust the network parameters of the preset adaptation model, and the performance of the preset adaptation model is continuously optimized and improved, and finally the performance and generalization ability of the video generation model are improved.

[0053] Step 305: Obtain a first video segment of the input video, wherein the first video segment includes doodle images of all video frames of the input video.

[0054] This step can refer to Figure 1 For the sake of brevity, step 101 is not described again here.

[0055] Step 306: input the preset Gaussian noise and the condition information guiding the generation of the video into the pre-trained video generation model to generate the target video, wherein the condition information at least includes the first video clip.

[0056] This step can refer to Figure 1 For the sake of brevity, step 102 is not described again here.

[0057] In summary, according to the technical solution of the embodiment of the present disclosure, by obtaining graffiti images of all video frames of the input video, and combining Gaussian noise and conditional information to guide video generation, a pre-trained video generation model is used to generate a target video, thereby realizing the process of automatically converting relatively rough line drawings such as graffiti images into animated videos, helping different users to achieve their video generation needs, improving the user's interactive experience, reducing the requirements for input information for animated video generation, and increasing the scope of applicable fields. In addition, by using graffiti images of all video frames of the input video, more image feature information can be provided to the pre-trained video generation model, thereby improving the matching degree between the target video and the graffiti images, and improving the generation quality of the target video.

[0058] For example, see Figure 4 As shown, in a video generation method of another embodiment of the method disclosed herein, a schematic diagram of video generation based on a pre-trained video generation model is shown. Figure 4As shown, the preset adaptation model takes the first video clip as input, firstly performs a decomposition on the image in the first video clip through its encoder (such as VAE variational autoencoder) to obtain the latent space expression of the first video clip, and the latent space expression of the first video clip is input into the preset adaptation model in the pre-trained video generation model to obtain the output vector of each computing layer in the preset adaptation model, and the output vector of each computing layer is saved. The random Gaussian noise and the target video prompt word are passed through the encoder to obtain the corresponding expression vector, wherein the target video prompt word is text information, and the corresponding text expression vector can be obtained by, for example, TextEncoder text encoding processing. The expression vectors corresponding to the random Gaussian noise and the target video prompt word are input into the preset video model. Since the multiple computing layers of the input layer of the preset video model and the multiple computing layers of the preset adaptation model are mapped one by one, the preset video model obtains the output vector of each computing layer in the preset adaptation model in each time step of the denoising process, and the output vector of each computing layer of the preset video model input layer is added to the output vector of the corresponding computing layer of the preset adaptation model, and the obtained calculation result is used as the input vector of the next computing layer in the input layer of the preset video model. And, the output vector of the preset video model output layer can be used as the input vector of the preset video model input layer of the next time step. It can be understood that after the denoising process of T time steps, the latent space expression of the target video is obtained, and the latent space expression of the target video can be obtained by the decoder to obtain the target video in pixel space.

[0059] Correspondingly, the embodiments of the present disclosure also provide device embodiments corresponding to the aforementioned method embodiments. Figure 5 This is a structural block diagram of a video generation device according to an embodiment of the present disclosure. Figure 5 As shown, a video generation device according to an embodiment of the present disclosure may include an image acquisition module 501 and a video generation module 502, wherein: An image acquisition module 501 is used to acquire a first video segment of an input video, wherein the first video segment includes graffiti images of all video frames of the input video; A video generation module 502, configured to input preset Gaussian noise and condition information guiding video generation into a pre-trained video generation model to generate a target video, wherein the condition information at least includes the first video clip; The pre-trained video generation model is jointly trained by a preset video model and a preset adaptation model.

[0060] In summary, according to the technical solution of the embodiment of the present disclosure, by obtaining graffiti images of all video frames of the input video, and combining Gaussian noise and conditional information to guide video generation, a pre-trained video generation model is used to generate a target video, thereby realizing the process of automatically converting relatively rough line drawings such as graffiti images into animated videos, helping different users to achieve their video generation needs, improving the user's interactive experience, reducing the requirements for input information for animated video generation, and increasing the scope of applicable fields. In addition, by using graffiti images of all video frames of the input video, more image feature information can be provided to the pre-trained video generation model, thereby improving the matching degree between the target video and the graffiti images, and improving the generation quality of the target video.

[0061] exist Figure 5 Based on the device embodiment shown, the present disclosure also provides another device embodiment. Figure 6 This is a structural block diagram of a video generation device according to another embodiment of the present invention. Figure 6 As shown, a video generation device according to an embodiment of the present disclosure may include an image acquisition module 501, a video generation module 502, a data acquisition module 503, a first training module 504, a second training module 505, a model determination module 506, a video cropping module 507, and a video annotation module 508. Among them: The data acquisition module 503 is used to acquire a plurality of second video segments corresponding to a plurality of training video samples, a plurality of video segments of each of the training video samples, and annotated description text corresponding to each of the video segments, wherein the second video segments include doodles of all video frames of the training video samples; A first training module 504, configured to input the plurality of second video clips into the preset adaptation model, and save an output vector of each layer of the preset adaptation model; The second training module 505 is used to input the annotation description text corresponding to each video segment in the multiple video segments of each training video sample into the preset video model in each round of training, obtain the predicted noise of the preset video model, and superimpose the output vector of the preset video model on the output vector of the input layer of each layer of the preset video model as the input vector of the next layer; A model determination module 506, configured to adjust the network parameters of the preset adaptation model based on the predicted noise of the preset video model to obtain the video generation model; The video cropping module 507 is used to crop each of the training video samples according to a preset resolution and a preset video segment duration to obtain a corresponding plurality of video segments; The video annotation module 508 is used to perform text annotation on each video clip using a preset video language model to obtain a corresponding annotation description text; An image acquisition module 501 is used to acquire a first video segment of an input video, wherein the first video segment includes graffiti images of all video frames of the input video; The video generation module 502 is used to input preset Gaussian noise and condition information guiding video generation into a pre-trained video generation model to generate a target video, wherein the condition information at least includes the first video clip.

[0062] As an optional embodiment of the present disclosure, the image acquisition module 501 may include: A video frame acquisition unit, used for acquiring a plurality of video frames of the input video according to a preset processing method; The scribble image extraction unit is used to extract the scribble image from each of the video frames to obtain the first video segment, wherein the timing of the scribble image in the first video segment is the same as the timing of the video frames in the input video.

[0063] As an optional embodiment of the present disclosure, the video generation module 502 may include: A first vector output unit, used to input the first video clip into a preset adaptation model of the pre-trained video generation model to obtain an output vector of each layer of the network architecture of the preset adaptation model; The target video determination unit is used to perform weighted calculation on the output vector of each layer of the input layer of the preset video model and the output vector of each layer of the preset adaptation model at each time step based on the preset Gaussian noise to obtain the target video.

[0064] As an optional embodiment of the present disclosure, the model determination module 506 may include: A network parameter fixing unit, used to fix the network parameters of each layer of the preset video model; The model generation unit is used to perform back propagation according to the loss function of the predicted noise calculation, and adjust the network parameters of each layer of the preset adaptation model to obtain the video generation model.

[0065] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present disclosure. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0066] Below, reference Figure 7 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0067] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0068] like Figure 7 As shown, the electronic device includes one or more processors and memory.

[0069] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0070] The memory may store one or more computer program products, and the memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program products may be stored on the computer-readable storage medium, and the processor may run the computer program product to implement the video generation method of each embodiment of the present disclosure described above and / or other desired functions.

[0071] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0072] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0073] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0074] Of course, to simplify, Figure 7 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0075] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the method of video generation according to various embodiments of the present disclosure described in the above part of this specification.

[0076] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0077] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps in the method of video generation according to various embodiments of the present disclosure described in the above part of this specification.

[0078] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0079] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0080] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0081] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0082] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0083] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0084] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0085] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for video generation, characterized in that: The method comprises: Obtaining a first video segment of an input video, wherein the first video segment includes doodle images of all video frames of the input video; Inputting preset Gaussian noise and condition information guiding video generation into a pre-trained video generation model to generate a target video, wherein the condition information at least includes the first video clip; The pre-trained video generation model is jointly trained by a preset video model and a preset adaptation model.

2. The method according to claim 1, characterized in that The step of obtaining a first video segment of an input video includes: Acquire multiple video frames of the input video according to a preset processing method; The scribble image is extracted from each of the video frames to obtain the first video segment, wherein the timing of the scribble image in the first video segment is the same as the timing of the video frames in the input video.

3. The method according to claim 1 or 2, characterized in that: The step of inputting the preset Gaussian noise and the condition information guiding the video generation into the pre-trained video generation model to generate the target video includes: Inputting the first video clip into a preset adaptation model of the pre-trained video generation model to obtain an output vector of each layer of the network architecture of the preset adaptation model; Based on the preset Gaussian noise, at each time step, the output vector of each layer of the input layer of the preset video model and the output vector of each layer of the preset adaptation model are weightedly calculated to obtain the target video.

4. The method according to claim 1, characterized in that: The method further comprises: Acquire multiple second video clips corresponding to multiple training video samples, multiple video clips of each of the training video samples, and annotated description text corresponding to each of the video clips, wherein the second video clips include doodles of all video frames of the training video samples; Inputting the plurality of second video clips into the preset adaptation model, and saving an output vector of each layer of the preset adaptation model; In each round of training, the annotated description text corresponding to each video segment in the multiple video segments of each training video sample is input into the preset video model to obtain the predicted noise of the preset video model, and the output vector of the preset video model is correspondingly superimposed on the output vector of the input layer of each layer of the preset video model as the input vector of the next layer; Adjust the network parameters of the preset adaptation model based on the predicted noise of the preset video model to obtain the video generation model; Among them, the input layer of the preset video model and the preset adaptation model have the same network architecture.

5. The method according to claim 4, characterized in that The step of adjusting the network parameters of the preset adaptation model based on the predicted noise of the preset video model to obtain the video generation model includes: Fixing the network parameters of each layer of the preset video model; Back propagation is performed according to the loss function of the predicted noise calculation, and the network parameters of each layer of the preset adaptation model are adjusted to obtain the video generation model.

6. The method according to claim 4 or 5, characterized in that: The method further comprises: Each of the training video samples is cropped according to a preset resolution and a preset video segment duration to obtain a corresponding plurality of video segments.

7. A video generation device, characterized in that: The device comprises: A scribble image acquisition module, used to acquire a first video segment of an input video, wherein the first video segment includes scribble images of all video frames of the input video; A video generation module, configured to input preset Gaussian noise and condition information guiding video generation into a pre-trained video generation model to generate a target video, wherein the condition information at least includes the first video clip; The pre-trained video generation model is jointly trained by a preset video model and a preset adaptation model.

8. An electronic device, characterized in that: include: a memory for storing a computer program product; A processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, it implements the method described in any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising computer program instructions, characterized in that When the computer program instructions are executed by a processor, the method described in any one of claims 1 to 6 is implemented.

Images