Video generation method, device and electronic equipment

Abstract

The present application relates to a video generation method, device and electronic device. The method includes: obtaining a video clip to be processed, the video clip includes multiple frames of reference images; inputting the video clip to be processed into a pre-trained adaptation module to obtain the fine-tuning vector corresponding to each first hidden layer of the adaptation module; the first hidden layer and the second hidden layer of the diffusion model have the same layer structure; inputting conditional information into the diffusion model to obtain the corresponding target video; the conditional information at least includes the video clip to be processed, and the output vector of each second hidden layer is used to superimpose the fine-tuning vector output by the mapped first hidden layer to serve as the input vector of the second hidden layer of the next layer; arranging the target images of each frame in time sequence to form a target video. The solution provided by the present application can be combined with a trained adaptation module to generate a target video based on the visual content of the reference image at a low cost through a pre-trained, universal diffusion model, thereby realizing the restoration or reconstruction of the visual content.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence processing technology, and in particular to a video generation method, device and electronic device. Background Art

[0002] With the rapid development of artificial intelligence technology, AIGC (Artificial Intelligent Generative Content) has been widely used in various fields. Among them, the application of image-based video generation is booming.

[0003] Current image-generated videos often use image-generated video models to generate short, simple videos based on one or several frames of high-definition images. However, current models are unable to generate high-quality videos from low-quality images (such as blurry images, line drawings, and black-and-white line drawings), or they are unable to restore or reconstruct the content of low-quality videos. Summary of the Invention

[0004] In order to solve or partially solve the problems existing in the related art, the present application provides a video generation method, device and electronic device, which can combine a trained adaptation module to generate a target video based on the visual content of a reference image at a low cost through a pre-trained, universal diffusion model, thereby realizing the restoration or reconstruction of the visual content.

[0005] The first aspect of the present application provides a video generation method, which includes:

[0006] Acquire a video clip to be processed, wherein the video clip includes multiple frames of reference images;

[0007] Inputting the video clip to be processed into a pre-trained adaptation module to obtain a fine-tuning vector corresponding to the first hidden layer of each layer of the adaptation module; wherein the first hidden layer has the same layer structure as the second hidden layer of the diffusion model;

[0008] Conditional information is input into the diffusion model to obtain a corresponding target video; wherein the conditional information includes at least the video clip to be processed, and the output vector of the second hidden layer of each layer is used to be superimposed with the fine-tuning vector output by the mapped first hidden layer to serve as the input vector of the second hidden layer of the next layer.

[0009] In some implementations, obtaining the video clip to be processed includes:

[0010] According to the preset video frame rate and preset duration, reference images of the corresponding number of frames are extracted in the same sample video in time sequence;

[0011] The reference images are arranged in time sequence to form the video segment to be processed.

[0012] In some implementations, the multiple frames of reference images are edge detection images or line drawings.

[0013] In some implementations, obtaining the video clip to be processed includes:

[0014] According to a preset video frame rate and a preset duration, reference images of a corresponding number of frames are extracted in a single sample video in a time sequence; and the reference images are arranged in a time sequence to form the video segment to be processed.

[0015] In some implementations, the condition information further includes prompt text;

[0016] Before inputting the condition information into the diffusion model to obtain the corresponding target video, the method further includes:

[0017] The prompt text is obtained based on a custom design and / or according to the visual content of a sample video for the intended purpose.

[0018] In some embodiments, the first hidden layer and the second hidden layer of the diffusion model have the same layer structure, including:

[0019] The number of layers in the first hidden layer of the adaptation module is equal to the number of layers in the second hidden layer and corresponds layer by layer;

[0020] The vector dimension of the first hidden layer is the same as the vector dimension of the second hidden layer of the corresponding level.

[0021] In some implementations, the adaptation module is trained according to the following method in combination with the pre-trained diffusion model:

[0022] Acquire multiple sets of training samples; wherein each set of training samples includes a training video and corresponding text annotations, and the training video includes multiple frames of edge detection images continuously extracted from the same sample video;

[0023] In each round of training, the training video is input into the adaptation module, and the fine-tuning vector output by the first hidden layer of each layer is saved respectively;

[0024] Inputting random noise and the training sample into the diffusion model to obtain a predicted noise image output by the diffusion model; wherein the output vector of the second hidden layer of each layer in the diffusion model is superimposed with the fine-tuning vector of the mapped first hidden layer as the input vector of the second hidden layer of the next layer;

[0025] The network parameters of each layer of the diffusion model are fixed, and the network parameters of the first hidden layer of each layer are adjusted according to the back propagation of the loss function of the diffusion model to obtain a trained adaptation module.

[0026] In some embodiments, the method further comprises:

[0027] Before training the adaptation module, the network parameters of the second hidden layer of each layer of the pre-trained diffusion model are used as the initialization parameters of the first hidden layer of each layer of the mapping of the adaptation module.

[0028] A second aspect of the present application provides a video generation device, comprising:

[0029] An image acquisition module is used to acquire a video clip to be processed, wherein the video clip includes multiple frames of reference images;

[0030] A model adaptation module, configured to input the video clip to be processed into a pre-trained adaptation module and obtain a fine-tuning vector corresponding to the first hidden layer of each layer of the adaptation module; wherein the first hidden layer has the same layer structure as the second hidden layer of the diffusion model;

[0031] A video generation module is configured to input conditional information into the diffusion model to obtain a corresponding target video; wherein the conditional information includes at least the video clip to be processed, and the output vector of the second hidden layer of each layer is used to be superimposed with the fine-tuning vector output by the mapped first hidden layer to serve as the input vector of the second hidden layer of the next layer.

[0032] A third aspect of the present application provides an electronic device, including:

[0033] processor; and

[0034] A memory stores executable code thereon, and when the executable code is executed by the processor, the processor is caused to execute the video generation method as described in the first aspect above.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor executes the video generation method described in the first aspect above.

[0036] A fifth aspect of the present application provides a computer program product, comprising a computer program, characterized in that the computer program is used to execute computer program code instructions corresponding to the video generation method described in the first aspect above.

[0037] The technical solution provided by this application may have the following beneficial effects:

[0038] The video generation method of this application uses a pre-trained adaptation module external to a general diffusion model to restore or reconstruct the visual content of a low-quality reference image in a video clip, combining conditional information. This design can produce a target video that is visually relevant to the reference image, meeting the user's video production needs.

[0039] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail the exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0041] Figure 1 This is a flow chart of a video generation method provided in an embodiment of the present application;

[0042] Figure 2 is a structural diagram of the adaptation module and diffusion model provided in an embodiment of the present application;

[0043] Figure 3 This is a flow chart of an adaptation module training method provided in an embodiment of the present application;

[0044] Figure 4 This is another flowchart of a video generation method provided by an embodiment of the present application;

[0045] Figure 5 is a structural diagram of a video generation device provided in an embodiment of the present application;

[0046] Figure 6 is another structural diagram of the video generating device provided in an embodiment of the present application;

[0047] Figure 7 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following describes embodiments of the present application in more detail with reference to the accompanying drawings. Although the accompanying drawings illustrate embodiments of the present application, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0049] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0050] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0051] Low-quality videos, such as those containing only lines or unclear images, affect the user's visual experience. In the current AIGC field, image-based video models are unable to restore or reconstruct low-quality videos.

[0052] To address the above problems, the present application provides a video generation method that can combine a trained adaptation module to generate videos based on low-quality reference images through a diffusion model at low cost, thereby achieving restoration or reconstruction of video content.

[0053] The technical solution of this application is described in detail below with reference to the accompanying drawings.

[0054] See also Figure 1 , a video generation method shown in this application includes:

[0055] S110 , obtaining a video clip to be processed, where the video clip includes multiple frames of reference images.

[0056] The video clip to be processed in the present application is composed of reference images of a corresponding number of frames according to a preset frame rate and duration. In some embodiments, the reference image is an edge detection image or a line drawing. Multiple frames of reference images can be obtained based on the original image in a sample video. It can be understood that the sample video is composed of multiple frames of original images arranged in time sequence, and the original image can be a line drawing or a color image. When the original image is a line drawing, each frame of the line drawing can be directly used as a reference image. When the original image is a color image, for example, it can be converted and processed according to the Canny edge detection method to form a corresponding edge detection image as a reference image. It can be understood that each frame of the reference image is arranged in sequence according to the corresponding time sequence in the sample video to guide the continuity of the subsequently generated video content.

[0057] Among them, the edge detection map includes the edge information in the original image, which facilitates the accurate identification of objects in the image. Therefore, on the basis of accurately capturing as many edges as possible in the image, it can subsequently provide the video generation model with more accurate information and generate a better target video.

[0058] S120 , inputting the video clip to be processed into a pre-trained adaptation module to obtain a fine-tuning vector corresponding to the first hidden layer of each layer of the adaptation module; wherein the first hidden layer has the same layer structure as the second hidden layer of the diffusion model.

[0059] The pre-trained adaptation module is obtained by joint training with the diffusion model. Optionally, in the joint training process, first: obtain the pre-trained diffusion model and the adaptation module to be trained, wherein the pre-trained diffusion model has the ability to generate videos, but its video generation effect is poor for specific scenes, and the network parameters of the pre-trained diffusion model are relatively large, such as billions or tens of billions. If the parameters of the pre-trained diffusion model are directly retrained and adjusted for specific scenes, it will not only affect the versatility of the pre-trained diffusion model, but also require very large training resources and a long cycle. The adaptation module includes a first hidden layer, and the diffusion model includes a second hidden layer (i.e., an input layer, for downsampling) and a third hidden layer (i.e., an output layer, for upsampling). In this embodiment, the first hidden layer of the adaptation module to be trained has the same layer structure as the second hidden layer of the diffusion model, that is, the number of layers of the first hidden layer of the adaptation module is the same as the number of layers of the network layer of the input layer of the diffusion model, and the first hidden layer of each adaptation module has the same network parameters as the network layer of the input layer of the corresponding diffusion model. Secondly: fix the network parameters of each layer of the diffusion model, and adjust the network parameters of each network layer of the adaptation module according to the back propagation of the loss function of the diffusion model, so as to obtain the pre-trained adaptation module.

[0060] In an embodiment of the present invention, the pre-trained adaptation module obtained through joint training with the diffusion model can avoid directly adjusting the parameters of the diffusion model, which not only reduces the number of parameters required for fine-tuning, but also reduces the demand for computing resources during training, thereby accelerating training. At the same time, when the diffusion model performs a specific task, the pre-trained adaptation module is used to fine-tune the vectors of the diffusion model's input layer, thereby achieving efficient and accurate completion of the specific task without requiring large-scale overall training of the diffusion model. It can be seen that the pre-trained adaptation module obtained through joint training with the diffusion model achieves the rapid generation of high-quality target videos with the visual content of video clips based on AIGC technology, using conditional information of video clips including multiple frames of reference images, without affecting the versatility of the diffusion model.

[0061] In an embodiment of the present invention, the diffusion model is a fusion architecture of Transformer and Diffusion. Specifically, the Transformer model architecture is a deep learning architecture that relies on the self-attention mechanism. It is essentially composed of an encoder and a decoder. After the encoder maps the input data to the latent space expression, the latent space expression of the target output corresponding to the input data is obtained through the Diffusion denoising process, and the latent space expression of the target output is mapped to the corresponding data space through the decoder to finally obtain the target output.

[0062] Furthermore, the diffusion model is a generative model based on a deep learning algorithm. It generates high-quality content through a process of gradual denoising and denoising. It performs well in generating images, videos, audio and other high-dimensional data, and has strong performance in generating tasks. In this application, the diffusion model is a pre-trained model. After the above-mentioned encoder maps the edge detection map to the latent space expression, under the guidance of the latent space expression of the edge detection map, the random noise is denoised for T time steps, that is, downsampling and upsampling are performed at each time step to obtain the latent space expression of the target video, and then the latent space expression of the target video is further mapped to the pixel space by the above-mentioned decoder to obtain the target video. Among them, the latent space expression of the edge detection map is the latent space expression obtained after encoding the edge detection map using the encoder. The encoder is responsible for mapping the data from the original data space to the latent space (Latent Space) to obtain an implicit and continuous representation, which can make the model calculation more efficient; and the decoder can reconstruct the video from this latent space expression and map the generation result of the diffusion model from the latent space to the pixel space. In some embodiments, the encoder and decoder used in the diffusion model of the present application can both be variational autoencoders (VAE). The latent space of VAE is continuous and suitable for generating continuous data, that is, the generated target videos or images are correlated.

[0063] For example, Figure 2As shown, the core module in the diffusion model is the U-Net network. U-Net is a structure of a downsampling module-upsampling module with jump connections, which is used to denoise random noise and realize the generation from noise to video or image. Among them, the downsampling module is used to gradually downsample the input noise, extract features and reduce the spatial resolution of the image. The upsampling module gradually restores the spatial resolution of the image through deconvolution (or transposed convolution) and upsampling operations; through jump connections, the upsampling module can combine the low-level features (such as edges, textures and other detail information) in the downsampling module with high-level features (such as semantic information of prompt text). The fusion of features helps to reconstruct the details of the image; and finally outputs a target image with the same resolution as the input image. In this application, based on the multi-layer second hidden layer structure in the downsampling module of the diffusion model, an independent adaptation module is designed to fine-tune the vector generated by the downsampling process. In other words, the adaptation module adapts to new tasks by inserting additional parameters into the pre-trained diffusion model, rather than directly adjusting the parameters of the diffusion model (the parameters of the diffusion model are on the order of billions or tens of billions). This not only reduces the number of parameters required for fine-tuning, but also reduces the demand for computing resources during training, speeding up training. At the same time, when the diffusion model performs a specific task, the trained adaptation module fine-tunes the output vectors of each layer of the second hidden layer, eliminating the need for large-scale overall training of the diffusion model, and enabling efficient and accurate completion of specific tasks. It can be seen from this that the combination of the adaptation module and the diffusion model enables the rapid generation of high-quality target videos based on AIGC technology using video clips that include edge detection maps, without affecting the versatility of the diffusion model.

[0064] like Figure 2As shown, the adaptation module has multiple first hidden layers arranged in sequence, and the downsampling module of the diffusion model has multiple second hidden layers arranged in sequence. In some embodiments, the number of first hidden layers in the adaptation module is less than or equal to the number of second hidden layers, and the layers correspond layer by layer. For example, if the adaptation module has M first hidden layers and the downsampling module has N second hidden layers, then 0 < M ≤ N, and both M and N are positive integers. Specifically, when M equals N, each first hidden layer is mapped one-to-one with the second hidden layer of the same level. When M is less than N, each first hidden layer can be mapped one-to-one with the second hidden layer of the same level, or one-to-one with the second hidden layer of a different level, for example, corresponding to the second hidden layer of an odd-numbered layer, an even-numbered layer, a designated layer, or a random layer. This design allows the input vector of at least one second hidden layer to be fine-tuned by superimposing a fine-tuning vector, thereby changing the corresponding output vector. This allows the general diffusion model to be influenced by the task-specific adaptation module, making it more suitable for processing specific tasks and promoting the achievement of the desired goal. Furthermore, the vector dimension of the first hidden layer is the same as the vector dimension of the second hidden layer of the corresponding level. This design facilitates the superposition of the fine-tuning vector output by the first hidden layer and the output vector of the corresponding second hidden layer in the same dimension, so as to achieve comprehensive fine-tuning of the output vector of the second hidden layer.

[0065] Before the video clip is fed into the adaptation module, it undergoes processing via an encoder to obtain its corresponding latent space representation. This latent space representation is then fed into the adaptation module, where the first hidden layers of each layer progressively downsample the latent space representation. The fine-tuned vector output by each first hidden layer serves as the input vector for the next first hidden layer, and the fine-tuned vectors output by each first hidden layer are stored.

[0066] S130: Inputting conditional information into the diffusion model to obtain a corresponding target video. The conditional information includes at least the video clip to be processed. The output vector of the second hidden layer of each layer is superimposed with the fine-tuning vector output by the mapped first hidden layer to serve as the input vector of the second hidden layer of the next layer.

[0067] It can be understood that after the conditional information is input into the diffusion model, the latent space expression corresponding to the conditional information is obtained through the processing of the encoder. The random noise is input into the diffusion model for denoising, and down-sampling is performed layer by layer through the second hidden layer of each layer of the diffusion model; during the downsampling process, the output vector of each second hidden layer is superimposed with the mapped fine-tuning vector to form the input vector of the next second hidden layer. And so on, until the output vector of the last second hidden layer is superimposed with the corresponding fine-tuning vector to form the final output vector of the downsampling module. The final output vector of the downsampling module is used as the input vector of the upsampling module for upsampling. Optionally, the output vector of the second hidden layer and the corresponding fine-tuning vector can be directly superimposed or weighted superimposed.

[0068] It can be understood that the above-mentioned downsampling module and upsampling module perform one round of calculation as a noise prediction for one time step, that is, a denoising process for one time step, and the diffusion model performs T rounds of iterative denoising to obtain the final target video. After the upsampling module outputs the output vector of the first round, it is input into the downsampling module again as the initial input vector of the second round. After T rounds of loop execution according to the above process, the upsampling module outputs the output vector of the Tth round to input the target video obtained by the decoder. During each round of iteration of the diffusion model, the second hidden layer of each layer of the downsampling module needs to be superimposed with the fine-tuning vector of the first hidden layer mapped by the above-mentioned adaptation module. The decoder can reconstruct the video from the latent space expression, map the generation result of the diffusion model from the latent space to the pixel space, and generate the target video.

[0069] In the present application, the conditional information at least includes the video clip to be processed. Based on this, the target video generated by the video generation model includes the visual content corresponding to the above-mentioned reference images of each frame. In other words, the conditional information includes the reference content expected by the user. Based on the guidance of the diffusion model by the reference content, a high-quality target video that meets the user's expectations can be generated. Specifically, each frame of the reference image is based on the existing visual content with continuous expression as a reference, so that the overall content of the target video is complete and reasonable. On the basis of the visual content corresponding to the reference image, the diffusion model, based on the learned knowledge, enables the generated target video to also include other illustration content, such as color, background image, etc., so that the target video content is more colorful.

[0070] As this example demonstrates, the video generation method of this application, by integrating a pre-trained adaptation module with a diffusion model, can restore or reconstruct the content of a low-quality reference image based on conditional information. This design can produce a target video that is relevant to the visual content of the reference image, meeting the user's video production needs.

[0071] See also Figure 3An embodiment of the present application further provides a training method for the adaptation module of the present application. The adaptation module of the present application can be obtained by training according to the following method.

[0072] S210, obtaining multiple groups of training samples; wherein each group of training samples includes a training video and corresponding text annotations, and the training video includes multiple frames of edge detection images continuously extracted from the same sample video.

[0073] In this application, some high-quality videos, such as videos with clear images and rich content, can be obtained as sample videos. According to the edge detection algorithm, the continuous multi-frame sample images of the sample video arranged in time sequence are converted to form an edge detection map, and the multi-frame edge detection map constitutes a corresponding training video in time sequence. Each training video is annotated with text annotations related to its visual content. By using high-quality sample videos to convert training videos, the adaptation module can learn richer features in subsequent training; at the same time, using edge detection maps with continuous and complete visual content as training samples, it is convenient for the adaptation module of subsequent training to learn the logical expression when restoring or reconstructing video content. Among them, the resolution, frame rate and duration of each training video can be consistent with the target video to be generated.

[0074] S220 , in each round of training, input the training video into the adaptation module, and save the fine-tuning vector output by the first hidden layer of each layer respectively.

[0075] In this step, the model architecture of the adaptation module is pre-set. Figure 2 As shown, a trained diffusion model with a corresponding network structure, U-Net, is selected, and the network structure of the adaptation module is set to be consistent with the network structure of the downsampling module in the diffusion model. In this embodiment, the number of layers in the first hidden layer of the adaptation module is consistent with the number of layers in the second hidden layer of the downsampling module of the diffusion model, and the vectors of the first hidden layer and the second hidden layer have the same dimension.

[0076] In some implementations, the network parameters of the adaptation module are initialized before training the adaptation module. Specifically, the network parameters of the second hidden layer of each layer in the pre-trained diffusion model are used as the initialization parameters of the first hidden layer of each layer mapped by the adaptation module.

[0077] After initializing the adaptation module, training officially begins. Each frame of the training video's edge detection image is encoded and mapped to a latent space representation by the encoder. This latent space representation is then fed into the adaptation module, and the first hidden layer of each layer outputs and stores the corresponding fine-tuning vector.

[0078] It should be noted that step S220 can be performed alternately with step S230, i.e., depending on the number of first hidden layers, step S220 can be performed once, followed by step S230, then step S220 again, and then step S230 again, until the last first hidden layer outputs a fine-tuning vector, at which point step S220 ends. Alternatively, step S220 can be performed first, with all fine-tuning vectors output by the first hidden layer being saved in sequence, before step S230 is performed.

[0079] S230, random noise and training samples are input into the diffusion model to obtain a predicted noise image output by the diffusion model; wherein the output vector of each second hidden layer in the diffusion model is superimposed with the fine-tuning vector of the mapped first hidden layer as the input vector of the second hidden layer of the lower layer.

[0080] Optionally, the diffusion model and the adaptation module can share an encoder or use separate encoders. In this step, each frame of the edge-detected image is randomly noised and input into the diffusion model along with the text annotations. Referring to the operating principle of the diffusion model in step S120, the random noise is denoised for T time steps using the conditional guidance of the edge-detected image to obtain a predicted noise image.

[0081] Taking the case where the number of layers of the first hidden layer is the same as the number of layers of the second hidden layer, at each time step, the output vector of the second hidden layer of the first layer is added to the fine-tuning vector of the output of the first hidden layer of the first layer obtained in S220 to obtain the input vector of the second hidden layer of the second layer. The output vector of the second hidden layer of the second layer is added to the fine-tuning vector of the output of the first hidden layer of the second layer to obtain the input vector of the second hidden layer of the third layer. This process is repeated until the output vector of the last hidden layer is superimposed with the fine-tuning vector of the last hidden layer to obtain the final output vector of the downsampling module. The vector is then fed into the diffusion model through forward propagation for upsampling. After T rounds of iterative denoising, the vector is input into the decoder, and the predicted noise image corresponding to each frame of the edge detection image is finally output.

[0082] S240, fixing the network parameters of each layer of the diffusion model, and adjusting the network parameters of the first hidden layer of each layer according to the back propagation of the loss function of the diffusion model to obtain a trained adaptation module.

[0083] In this application, the training process iteratively adjusts the network parameters of the adaptation module, while the network parameters of each layer of the diffusion model remain fixed. Based on this, after calculating the noise loss function between the predicted noise image and the true value pixels of the edge detection image, the network parameters of the first hidden layer of each layer of the adaptation module are adjusted by backpropagation. This loop is iterated from S220 to S240 until the loss function converges, resulting in a trained adaptation module with the trained network parameters of each first hidden layer.

[0084] From this example, it can be seen that the training method of the present application can be based on a small amount of high-quality sample videos, and the edge detection map of the continuous expression of the content can be converted as a training video, and then the adaptation module can be connected to the pre-trained diffusion model to realize the training of the adaptation module, thereby efficiently and at low cost obtaining a diffusion model that can restore or reconstruct the video content and is connected to the adaptation module.

[0085] See also Figure 4 According to the combination of the trained adaptation module and the pre-trained diffusion model in the above embodiment, the video generation method of the present application will be further introduced in conjunction with the embodiment below, which includes:

[0086] S310 , extracting reference images of a corresponding number of frames in a single sample video in time sequence according to a preset video frame rate and a preset duration; and arranging the reference images in time sequence to form a video segment to be processed.

[0087] Users can select an existing sample video to generate a video clip to be processed. Optionally, the sample video's frame rate can be greater than or equal to the frame rate of the video clip to be processed, and the sample video's duration can be greater than or equal to the duration of the video clip to be processed. To obtain a video clip that meets the preset frame rate and duration, multiple continuous reference frames that meet the preset duration and frame rate can be captured from the sample video.

[0088] If the original image in the sample video consists of a line drawing or an edge detection image, the original image is directly used as the reference image required for the subsequent steps. If the original image in the sample video is not an edge detection image, it can be converted into an edge detection image using a relevant edge detection algorithm and then used as the input image for the subsequent steps S320 and S330.

[0089] S320 , inputting the video clip to be processed into a pre-trained adaptation module to obtain a fine-tuning vector corresponding to the first hidden layer of each layer of the adaptation module; wherein the first hidden layer has the same number of layers and layer structure as the second hidden layer of the diffusion model.

[0090] This step is the same as the introduction of S120 and will not be repeated here.

[0091] S330, input the video clip and prompt text into the pre-trained diffusion model to obtain the corresponding target video; wherein the output vector of the second hidden layer of each layer is used to be superimposed with the fine-tuning vector output by the mapped first hidden layer to serve as the input vector of the second hidden layer of the next layer.

[0092] In this embodiment, the conditional information also includes prompt text. The prompt text includes at least attribute information of the subject object in the target video, for example, the type of the subject object and other attribute information of the subject object. The type of the subject object may include: human type, animal type, or other object type (such as an object that needs to be moved in a movie or game scene). Other attribute information of the subject object may include color information of the subject object, structural information (such as the body proportions of the subject object when the subject object is a person or animal), and other information.

[0093] As a general video generation model, the diffusion model has a relatively strong ability to understand text. In this embodiment, by taking the prompt text as the input of the diffusion model, it can be ensured that the generated target video better meets the user's expectations.

[0094] Based on the intended goal, prompt text is generated based on custom design and / or the visual content of the sample video. When the intended goal is to restore the video content, the prompt text should be as consistent as possible with the visual content of the sample video clip. When the intended goal is to reconstruct the video content, the prompt text can be different from the visual content of the sample video.

[0095] Referring to the description of S130 above, the fine-tuning vector output by each layer of S320 is superimposed with the output vector of the second hidden layer of the corresponding level to obtain a new vector with the same vector dimension but changed vector value, which is then input into the second hidden layer of the next level. Specifically, after random noise is added to each reference image in the video clip to be processed, it is input into the diffusion model together with the prompt text. According to the latent space expression corresponding to each frame of the encoded input image, each first hidden layer in the adaptation module outputs the corresponding fine-tuning vector Δd one by one; for example, the first hidden layer of the first layer outputs a fine-tuning vector Δd1, the first hidden layer of the second layer outputs a fine-tuning vector Δd2, and the first hidden layer of the nth layer outputs a fine-tuning vector Δd n .

[0096] Correspondingly, the output vector of the second hidden layer of the first layer in the diffusion model is d1. After superimposing d1 and Δd1, (d1+Δd1) is used as the input vector of the second hidden layer of the second layer. The output vector of the second hidden layer of the second layer is d2. After superimposing d2 and Δd2, (d2+Δd2) is used as the input vector of the second hidden layer of the third layer. Similarly, the output vector of the second hidden layer of the last layer is d n and Δd n After superposition, the final output vector is formed and input into the upsampling module for upsampling. The downsampling and upsampling modules perform denoising based on the prompt information of the prompt text to obtain a latent space representation that matches the visual content of the target video. After T rounds of iterative denoising, the vector is input into the decoder to obtain the target video.

[0097] The video generation method of the present application fine-tunes the denoising process of the diffusion model based on a pre-trained adaptation module, so that the diffusion model can obtain the denoising result influenced by the adaptation module according to the prompt text, and realize the restoration or reconstruction of the content of multiple frames of continuous reference images, thereby combining the generated target images to form a target video to meet user needs.

[0098] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides a video generation device and corresponding embodiments.

[0099] Figure 5 It is a structural diagram of the video generation device shown in this application.

[0100] See also Figure 5 The video generation device shown in this application includes an image acquisition module 410, a model adaptation module 420 and a video generation module 430. Among them:

[0101] The image acquisition module 410 is used to acquire a video segment to be processed, where the video segment includes multiple frames of reference images.

[0102] The model adaptation module 420 is used to input the video clip to be processed into the pre-trained adaptation module to obtain the fine-tuning vector corresponding to the first hidden layer of each layer of the adaptation module; wherein the first hidden layer has the same layer structure as the second hidden layer of the diffusion model.

[0103] The video generation module 430 is used to input conditional information into the diffusion model to obtain the corresponding target video; wherein the output vector of the second hidden layer of each layer is used to be superimposed with the fine-tuning vector output by the mapped first hidden layer to serve as the input vector of the second hidden layer of the next layer.

[0104] See also Figure 6 In a specific embodiment, the image acquisition module 430 is used to extract reference images of a corresponding number of frames in a single sample video in a time sequence according to a preset video frame rate and a preset duration; and arrange the reference images in a time sequence to form a video segment to be processed.

[0105] When the original image in the sample video is not an edge detection image, the apparatus of the present application further includes an edge detection module 440 for converting the original images of the corresponding number of frames in the sample video into an edge detection image.

[0106] From this example, it can be seen that the video generation device of the present application can insert an adaptation module into a pre-trained diffusion model, generate additional parameters through the adaptation module to adjust the downsampling process of the diffusion model, and while retaining the general knowledge of the diffusion model, it can adapt to the content production of new videos and restore or reconstruct the content of low-quality reference images as much as possible in combination with prompt text.

[0107] An embodiment of the present application further provides a training device for an adaptation module, which includes:

[0108] The sample acquisition module is used to obtain multiple groups of training samples; each group of training samples includes a training video and corresponding text annotations, and the training video contains multiple frames of edge detection images continuously extracted from the same sample video.

[0109] The fine-tuning vector module is used to input the training video into the adaptation module in each round of training and save the fine-tuning vector output by the first hidden layer of each layer.

[0110] The model training module is used to input random noise and training samples into the diffusion model to obtain the predicted noise image output by the diffusion model; wherein the output vector of the second hidden layer of each layer in the diffusion model is superimposed with the fine-tuning vector of the mapped first hidden layer as the input vector of the second hidden layer of the lower layer.

[0111] The parameter iteration module is used to fix the network parameters of each layer of the diffusion model, and adjust the network parameters of the first hidden layer of each layer according to the back propagation of the loss function of the diffusion model to obtain a trained adaptation module.

[0112] The adaptation module training device further includes an initialization module for using the pre-trained network parameters of the second hidden layer of each diffusion model as the initialization parameters of the first hidden layer of each layer before the fine-tuning vector module training.

[0113] The adaptation module training device of the present application adapts to specific downstream tasks by inserting a small, trainable adaptation module into a pre-trained diffusion model without fine-tuning the entire diffusion model. It allows the diffusion model to adapt to new tasks by adjusting a small number of parameters while keeping the original pre-trained parameters unchanged, thereby improving parameter efficiency and training speed.

[0114] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0115] Figure 7 It is a schematic structural diagram of the electronic device shown in this application.

[0116] See also Figure 7 , the electronic device 1000 includes a memory 1010 and a processor 1020.

[0117] The processor 1020 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0118] The memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage. ROM may store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage may be a readable and writable storage device. The permanent storage may be a non-volatile storage device that retains stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (e.g., a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (e.g., a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all instructions and data required by the processor during operation. In addition, the memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0119] The memory 1010 stores executable codes. When the executable codes are processed by the processor 1020 , the processor 1020 may execute part or all of the above-mentioned methods.

[0120] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0121] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0122] The embodiments of the present application have been described above. The above description is illustrative and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to the technology in the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein.

Claims

1. A video generation method, characterized in that: include: Obtain a video clip to be processed, wherein the video clip includes multiple frames of reference images; the multiple frames of reference images are edge detection images or line drawings and are obtained based on original images in a sample video; Inputting the video clip to be processed into a pre-trained adaptation module to obtain a fine-tuning vector corresponding to the first hidden layer of each layer of the adaptation module; wherein the first hidden layer has the same layer structure as the second hidden layer of the diffusion model; and the second hidden layer belongs to the downsampling module of the diffusion model; Conditional information is input into the diffusion model to obtain a corresponding target video; wherein the conditional information includes at least the video clip to be processed, and the output vector of each second hidden layer is used to be superimposed with the fine-tuning vector output by the mapped first hidden layer to serve as the input vector of the second hidden layer of the next layer.

2. The method according to claim 1, characterized in that The step of obtaining the video clip to be processed includes: According to the preset video frame rate and preset duration, reference images of the corresponding number of frames are extracted in time sequence from a single sample video; The reference images are arranged in time sequence to form the video segment to be processed.

3. The method according to claim 1, wherein: The condition information also includes prompt text; Before inputting the condition information into the diffusion model to obtain the corresponding target video, the method further includes: The prompt text is obtained based on a custom design and / or according to the visual content of a sample video for the intended purpose.

4. The method according to claim 1, wherein The first hidden layer has the same layer structure as the second hidden layer in the diffusion model, including: The number of layers in the first hidden layer of the adaptation module is equal to the number of layers in the second hidden layer and corresponds layer by layer; The vector dimension of the first hidden layer is the same as the vector dimension of the second hidden layer of the corresponding level.

5. The method according to claim 1, wherein Combined with the pre-trained diffusion model, the adaptation module is trained according to the following method: Acquire multiple sets of training samples; wherein each set of training samples includes a training video and corresponding text annotations, and the training video includes multiple frames of edge detection images continuously extracted from the same sample video; In each round of training, the training video is input into the adaptation module, and the fine-tuning vector output by the first hidden layer of each layer is saved respectively; Inputting random noise and the training sample into the diffusion model to obtain a predicted noise image output by the diffusion model; wherein the output vector of the second hidden layer of each layer in the diffusion model is superimposed with the fine-tuning vector of the mapped first hidden layer as the input vector of the second hidden layer of the next layer; The network parameters of each layer of the diffusion model are fixed, and the network parameters of the first hidden layer of each layer are adjusted according to the back propagation of the loss function of the diffusion model to obtain a trained adaptation module.

6. The method according to claim 5, characterized in that The method further comprises: Before training the adaptation module, the network parameters of the second hidden layer of each layer in the pre-trained diffusion model are used as the initialization parameters of the first hidden layer of each layer mapped by the adaptation module.

7. A video generating device, characterized in that: include: An image acquisition module is used to acquire a video clip to be processed, wherein the video clip includes multiple frames of reference images; the multiple frames of reference images are edge detection images or line drawings and are obtained based on original images in a sample video; A model adaptation module is used to input the video clip to be processed into a pre-trained adaptation module to obtain the fine-tuning vector corresponding to the first hidden layer of each layer of the adaptation module; wherein the first hidden layer has the same layer structure as the second hidden layer of the diffusion model; the second hidden layer belongs to the downsampling module of the diffusion model; a video generation module is used to input conditional information into the diffusion model to obtain the corresponding target video; wherein the conditional information includes at least the video clip to be processed, and the output vector of the second hidden layer of each layer is used to be superimposed with the fine-tuning vector output by the mapped first hidden layer to serve as the input vector of the second hidden layer of the next layer.

8. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the video generation method according to any one of claims 1 to 6.

9. A computer-readable storage medium having executable codes stored thereon, wherein when the executable codes are executed by a processor of an electronic device, the processor is caused to execute the video generation method according to any one of claims 1 to 6.

Images