Video generation method and device and electronic equipment

Abstract

The invention relates to a video generation method and device and electronic equipment. The method comprises the following steps: acquiring a video clip to be processed, wherein the video clip comprises multiple frames of reference images; inputting a to-be-processed video clip into a pre-trained adaptation module to obtain a fine tuning vector corresponding to each first hidden layer of the adaptation module; the first hidden layer and a second hidden layer of the diffusion model have the same layer structure; inputting the condition information into a diffusion model to obtain a corresponding target video; the condition information at least comprises a to-be-processed video clip, and an output vector of each second hidden layer is used for being overlapped with a fine adjustment vector output by the mapped first hidden layer to serve as an input vector of the next second hidden layer; and arranging the frames of target images according to a time sequence to form a target video. According to the scheme provided by the invention, the target video can be generated at low cost through the pre-trained and universal diffusion model based on the visual content of the reference image in combination with the trained adaptation module, and the restoration or reconstruction of the visual content is realized.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence processing technology, and in particular to a video generation method, device and electronic device. Background Art

[0002] With the rapid development of artificial intelligence technology, AIGC (Artificial Intelligent Generative Content) has been widely used in various fields. Among them, the application of generating videos based on images is booming.

[0003] The common scenario of current image-generated videos is to generate a short video with a single content based on a given frame or several frames of high-definition images. However, the current model cannot generate a high-quality video based on some low-quality images (such as unclear images, line drawings, black and white line drawings, etc.), or cannot restore or reconstruct the video content of low-quality videos. Summary of the invention

[0004] In order to solve or partially solve the problems existing in the related art, the present application provides a video generation method, device and electronic device, which can combine a trained adaptation module to generate a target video based on the visual content of a reference image at a low cost through a pre-trained, general diffusion model, thereby realizing the restoration or reconstruction of the visual content.

[0005] The first aspect of the present application provides a video generation method, which includes:

[0006] Acquire a video clip to be processed, wherein the video clip includes multiple frames of reference images;

[0007] Input the video clip to be processed into a pre-trained adaptation module to obtain a fine-tuning vector corresponding to each first hidden layer of the adaptation module; wherein the first hidden layer has the same layer structure as the second hidden layer of the diffusion model;

[0008] The conditional information is input into the diffusion model to obtain the corresponding target video; wherein the conditional information at least includes the video clip to be processed, and the output vector of the second hidden layer of each layer is used to be superimposed with the fine-tuning vector output by the mapped first hidden layer to serve as the input vector of the second hidden layer of the next layer.

[0009] In some implementation modes, obtaining the video clip to be processed includes:

[0010] According to a preset video frame rate and a preset duration, reference images of a corresponding number of frames are extracted in time sequence from the same sample video;

[0011] The reference images are arranged in time sequence to form the video segment to be processed.

[0012] In some implementations, the multiple frames of reference images are edge detection images or line drawings.

[0013] In some implementation modes, obtaining the video clip to be processed includes:

[0014] According to a preset video frame rate and a preset duration, reference images of a corresponding number of frames are extracted in a single sample video in time sequence; and the reference images are arranged in time sequence to form the video segment to be processed.

[0015] In some implementations, the condition information further includes prompt text;

[0016] Before inputting the condition information into the diffusion model to obtain the corresponding target video, the method further includes:

[0017] The prompt text is obtained based on a custom design and / or according to the visual content of a sample video for the intended purpose.

[0018] In some implementations, the first hidden layer and the second hidden layer of the diffusion model have the same layer structure, including:

[0019] The number of layers of the first hidden layer of the adaptation module is equal to the number of layers of the second hidden layer and corresponds layer by layer;

[0020] The vector dimension of the first hidden layer is the same as the vector dimension of the second hidden layer of the corresponding level.

[0021] In some implementations, the adaptation module is obtained by training in combination with the pre-trained diffusion model according to the following method:

[0022] Acquire multiple groups of training samples; wherein each group of training samples includes a training video and corresponding text annotations, and the training video includes multiple frames of edge detection images continuously extracted from the same sample video;

[0023] In each round of training, the training video is input into the adaptation module, and the fine-tuning vector output by the first hidden layer of each layer is saved respectively;

[0024] Inputting random noise and the training sample into the diffusion model to obtain a predicted noise image output by the diffusion model; wherein the output vector of each second hidden layer in the diffusion model is superimposed with the fine-tuning vector of the first hidden layer mapped as the input vector of the second hidden layer of the lower layer;

[0025] The network parameters of each layer of the diffusion model are fixed, and the network parameters of the first hidden layer of each layer are adjusted according to the back propagation of the loss function of the diffusion model to obtain a trained adaptation module.

[0026] In some embodiments, the method further comprises:

[0027] Before training the adaptation module, the network parameters of the second hidden layers of each layer of the pre-trained diffusion model are used as initialization parameters of the first hidden layers of each layer of the mapping of the adaptation module.

[0028] A second aspect of the present application provides a video generation device, comprising:

[0029] An image acquisition module, used to acquire a video clip to be processed, wherein the video clip includes multiple frames of reference images;

[0030] A model adaptation module, used for inputting the video clip to be processed into a pre-trained adaptation module to obtain a fine-tuning vector corresponding to each first hidden layer of the adaptation module; wherein the first hidden layer has the same layer structure as the second hidden layer of the diffusion model;

[0031] A video generation module is used to input conditional information into the diffusion model to obtain a corresponding target video; wherein the conditional information at least includes the video clip to be processed, and the output vector of each second hidden layer is used to superimpose the fine-tuning vector output by the mapped first hidden layer to serve as the input vector of the second hidden layer of the next layer.

[0032] A third aspect of the present application provides an electronic device, including:

[0033] Processor; and

[0034] A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the video generating method as described in the first aspect above.

[0035] A fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor executes the video generation method described in the first aspect above.

[0036] A fifth aspect of the present application provides a computer program product, including a computer program, characterized in that the computer program is used to execute computer program code instructions corresponding to the video generation method described in the first aspect above.

[0037] The technical solution provided by this application may have the following beneficial effects:

[0038] The video generation method of the present application can restore or reconstruct the visual content of the low-quality reference image of the video clip by combining conditional information by connecting a pre-trained adaptation module to a general diffusion model. Such a design can obtain a target video related to the visual content of the reference image, meeting the user's video production needs.

[0039] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail the exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0041] Figure 1 It is a flowchart of a video generation method provided in an embodiment of the present application;

[0042] Figure 2 is a schematic diagram of the structure of the adaptation module and the diffusion model provided in the embodiment of the present application;

[0043] Figure 3 It is a flowchart of an adaptation module training method provided in an embodiment of the present application;

[0044] Figure 4 is another flow chart of a video generation method provided by an embodiment of the present application;

[0045] Figure 5 is a schematic diagram of the structure of a video generating device provided in an embodiment of the present application;

[0046] Figure 6 is another structural schematic diagram of the video generating device provided in an embodiment of the present application;

[0047] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0049] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0050] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0051] For low-quality videos, such as those containing only lines or unclear images, which affect the user's visual experience, in the current AIGC field, the image-generated video model cannot restore or reconstruct low-quality videos.

[0052] In response to the above problems, the present application provides a video generation method that can combine a trained adaptation module to generate videos based on low-quality reference images through a diffusion model at a low cost, thereby achieving restoration or reconstruction of video content.

[0053] The technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0054] See also Figure 1 , a video generation method shown in the present application includes:

[0055] S110, obtaining a video segment to be processed, where the video segment includes multiple frames of reference images.

[0056] The video clip to be processed in the present application is composed of reference images of a corresponding number of frames according to a preset frame rate and duration. In some embodiments, the reference image is an edge detection image or a line drawing. Multiple frames of reference images can be obtained based on the original image in a sample video. It can be understood that the sample video is composed of multiple frames of original images arranged in time sequence, and the original image can be a line drawing or a color image. When the original image is a line drawing, each frame of the line drawing can be directly used as a reference image. When the original image is a color image, for example, it can be converted and processed according to the Canny edge detection method to form a corresponding edge detection image as a reference image. It can be understood that each frame of the reference image is arranged in sequence according to the corresponding time sequence in the sample video to guide the continuity of the subsequently generated video content.

[0057] Among them, the edge detection map includes the edge information in the original image, which facilitates the accurate identification of objects in the image. Therefore, on the basis of accurately capturing as many edges as possible in the image, it can subsequently provide the video generation model with more accurate information and generate a better target video.

[0058] S120, inputting the video clip to be processed into a pre-trained adaptation module to obtain a fine-tuning vector corresponding to each first hidden layer of the adaptation module; wherein the first hidden layer has the same layer structure as the second hidden layer of the diffusion model.

[0059] The pre-trained adaptation module is obtained by joint training with the diffusion model. Optionally, in the process of joint training, first: obtain the pre-trained diffusion model and the adaptation module to be trained, wherein the pre-trained diffusion model has the ability to generate videos, but its video generation effect is poor for specific scenes, and the network parameter magnitude of the pre-trained diffusion model is relatively large, such as billions or tens of billions. If the parameters of the pre-trained diffusion model are directly retrained and adjusted for specific scenes, it will not only affect the versatility of the pre-trained diffusion model, but also require very large training resources and a long cycle. The adaptation module includes a first hidden layer, and the diffusion model includes a second hidden layer (i.e., an input layer, used for downsampling) and a third hidden layer (i.e., an output layer, used for upsampling). In this embodiment, the first hidden layer of the adaptation module to be trained has the same layer structure as the second hidden layer of the diffusion model, that is, the number of layers of the first hidden layer of the adaptation module is the same as the number of layers of the network layer of the input layer of the diffusion model, and the first hidden layer of each adaptation module has the same network parameters as the network layer of the input layer of the corresponding diffusion model. Secondly: fix the network parameters of each layer of the diffusion model, and adjust the network parameters of each network layer of the adaptation module according to the back propagation of the loss function of the diffusion model, so as to obtain the pre-trained adaptation module.

[0060] In an embodiment of the present invention, the pre-trained adaptation module obtained by joint training with the diffusion model does not need to directly adjust the parameters of the diffusion model, which can not only reduce the number of parameters required for fine-tuning, but also reduce the demand for computing resources during training and speed up the training. At the same time, when the diffusion model performs a specific task, the vector of the input layer of the diffusion model is fine-tuned through the pre-trained adaptation module, so that the specific task can be completed efficiently and accurately without large-scale overall training of the diffusion model. It can be seen that the pre-trained adaptation module obtained by joint training with the diffusion model realizes the rapid generation of high-quality target videos with visual content of video clips based on AIGC technology, through the conditional information of video clips including multiple frames of reference images, and does not affect the versatility of the diffusion model.

[0061] In an embodiment of the present invention, the diffusion model is a fusion architecture of Transformer and Diffusion. Specifically, the Transformer model architecture is a deep learning architecture that relies on the self-attention mechanism, and is essentially composed of an encoder and a decoder. After the encoder maps the input data to the latent space expression, the latent space expression of the target output corresponding to the input data is obtained through the Diffusion denoising process, and the latent space expression of the target output is mapped to the corresponding data space through the decoder, and finally the target output is obtained.

[0062] Furthermore, the diffusion model is a generative model based on a deep learning algorithm, which generates high-quality content through a process of gradual denoising and denoising, performs well in generating images, videos, audio and other high-dimensional data, and has strong performance in generating tasks. In this application, the diffusion model is a pre-trained model. After the above-mentioned encoder maps the edge detection map to the latent space expression, under the guidance of the latent space expression of the edge detection map, the random noise is denoised for T time steps, that is, downsampling and upsampling are performed at each time step to obtain the latent space expression of the target video, and then the latent space expression of the target video is further mapped to the pixel space through the above-mentioned decoder to obtain the target video. Among them, the latent space expression of the edge detection map is the latent space expression obtained after encoding the edge detection map using the encoder. The encoder is responsible for mapping the data from the original data space to the latent space (Latent Space) to obtain an implicit and continuous representation, which can make the model calculation more efficient; and the decoder can reconstruct the video from this latent space expression and map the generation result of the diffusion model from the latent space to the pixel space. In some embodiments, the encoder and decoder used in the diffusion model of the present application can both be variational autoencoders (VAE). The latent space of VAE is continuous and suitable for generating continuous data, that is, the generated target videos or images are correlated.

[0063] For example, Figure 2As shown, the core module in the diffusion model is the U-Net network. U-Net is a structure of a downsampling module-upsampling module with jump connections, which is used to denoise random noise and realize the generation from noise to video or image. Among them, the downsampling module is used to gradually downsample the input noise, extract features and reduce the spatial resolution of the image. The upsampling module gradually restores the spatial resolution of the image through deconvolution (or transposed convolution) and upsampling operations; through jump connections, the upsampling module can combine low-level features (such as edge, texture and other detail information) in the downsampling module with high-level features (such as semantic information of prompt text), and the fusion of features helps to reconstruct the details of the image; finally, the target image with the same resolution as the input image is output. In this application, based on the multi-layer second hidden layer structure in the downsampling module of the diffusion model, an independent adaptation module is designed to fine-tune the vector generated by the downsampling process. That is to say, the adaptation module adapts to new tasks by inserting additional parameters into the pre-trained diffusion model, rather than directly adjusting the parameters of the diffusion model (the parameters of the diffusion model are in the order of billions or tens of billions). This can not only reduce the number of parameters required for fine-tuning, but also reduce the demand for computing resources during training, thereby speeding up training. At the same time, when the diffusion model performs a specific task, the output vectors of each layer of the second hidden layer are fine-tuned through the trained adaptation module, so that the diffusion model does not need to be trained on a large scale as a whole, and the specific task can be completed efficiently and accurately. It can be seen from this that through the combination of the adaptation module and the diffusion model, it is possible to quickly generate high-quality target videos based on AIGC technology through video clips including edge detection maps, without affecting the versatility of the diffusion model.

[0064] like Figure 2As shown, the adaptation module has multiple layers of first hidden layers arranged in sequence, and the downsampling module of the diffusion model has multiple layers of second hidden layers arranged in sequence. In some embodiments, the number of layers of the first hidden layer of the adaptation module is less than or equal to the number of layers of the second hidden layer and corresponds layer by layer. For example, the adaptation module has M layers of first hidden layers, and the downsampling module has N layers of second hidden layers, then 0<M≤N, and M and N are both positive integers. Specifically, when M is equal to N, each layer of the first hidden layer is mapped one-to-one with the second hidden layer of the same level. When M is less than N, each layer of the first hidden layer can be mapped one-to-one with the second hidden layer of the same level, or one-to-one with the second hidden layer of different levels, for example, with the second hidden layer corresponding to the odd layer, even layer, specified layer or random layer. Such a design allows the input vector of at least one second hidden layer to be fine-tuned by superimposing the fine-tuning vector, and then changes the corresponding output vector, so that the general diffusion model is affected by the adaptation module of the specific task, and is more inclined to the processing of the specific task to promote the achievement of the expected goal. Furthermore, the vector dimension of the first hidden layer is the same as the vector dimension of the second hidden layer of the corresponding level. Such a design facilitates the same-dimensional superposition of the fine-tuning vector output by the first hidden layer and the corresponding output vector of the second hidden layer, so as to achieve comprehensive fine-tuning of the output vector of the second hidden layer.

[0065] Before the video clip to be processed is input into the adaptation module, the latent space expression corresponding to the video clip is obtained through the processing of the encoder. The latent space expression of the video clip to be processed is input into the adaptation module, and the first hidden layer of each layer of the adaptation module gradually downsamples the latent space expression; the fine-tuning vector output by each first hidden layer is used as the input vector of the first hidden layer of the next layer, and the fine-tuning vector output by each first hidden layer is saved.

[0066] S130, inputting the conditional information into the diffusion model to obtain the corresponding target video, wherein the conditional information at least includes the video clip to be processed, and the output vector of each second hidden layer is used to be superimposed with the fine-tuning vector output by the mapped first hidden layer to serve as the input vector of the second hidden layer of the next layer.

[0067] It can be understood that after the conditional information is input into the diffusion model, the latent space expression corresponding to the conditional information is obtained through the processing of the encoder. The random noise is input into the diffusion model for denoising, and down-sampled layer by layer through the second hidden layers of each layer of the diffusion model; in the down-sampling process, the output vector of each second hidden layer is superimposed with the mapped fine-tuning vector to form the input vector of the next second hidden layer. And so on, until the output vector of the last second hidden layer is superimposed with the corresponding fine-tuning vector to form the final output vector of the down-sampling module. The final output vector of the down-sampling module is used as the input vector of the up-sampling module for up-sampling. Optionally, the output vector of the second hidden layer and the corresponding fine-tuning vector can be directly superimposed or weighted superimposed.

[0068] It can be understood that the above-mentioned downsampling module and upsampling module perform one round of calculation as a noise prediction for one time step, that is, a denoising process for one time step, and the diffusion model performs T rounds of iterative denoising to obtain the final target video. After the upsampling module outputs the output vector of the first round, it is input into the downsampling module again as the initial input vector of the second round. After T rounds of loop execution according to the above process, the upsampling module outputs the output vector of the Tth round to input the decoder to obtain the target video. In each round of iteration of the diffusion model, the second hidden layer of each layer of the downsampling module needs to be superimposed with the fine-tuning vector of the first hidden layer mapped by the above-mentioned adaptation module. The decoder can reconstruct the video from the latent space expression, map the generation result of the diffusion model from the latent space to the pixel space, and generate the target video.

[0069] In the present application, the conditional information at least includes the video clip to be processed. Based on this, the target video generated by the video generation model includes the visual content corresponding to the above-mentioned reference images. In other words, the conditional information includes the reference content expected by the user. Based on the guidance of the diffusion model by the reference content, a high-quality target video that meets the user's expectations can be generated. Specifically, each frame of the reference image is based on the existing visual content with continuous expression as a reference, so that the overall content of the target video is complete and reasonable. On the basis of the visual content corresponding to the reference image, the diffusion model, based on the learned knowledge, enables the generated target video to also include other illustration content, such as color, background image, etc., so that the target video content is more colorful.

[0070] From this example, we can see that the video generation method of the present application can restore or reconstruct the content of the low-quality reference image by connecting a pre-trained adaptation module to the diffusion model in combination with conditional information. Such a design can obtain a target video related to the visual content of the reference image, meeting the user's video production needs.

[0071] See also Figure 3An embodiment of the present application also provides a training method for the adaptation module of the present application. The adaptation module of the present application can be obtained by training according to the following method.

[0072] S210, obtaining multiple groups of training samples; wherein each group of training samples includes a training video and corresponding text annotations, and the training video includes multiple frames of edge detection images continuously extracted from the same sample video.

[0073] In the present application, some high-quality videos, such as videos with clear images and rich content, can be obtained as sample videos. According to the edge detection algorithm, the continuous multiple frames of sample images arranged in time sequence of the sample video are converted to form an edge detection map, and the multiple frames of edge detection maps form a corresponding training video in time sequence. Each training video is annotated with text annotations related to its visual content. By using high-quality sample videos to convert training videos, the adaptation module can learn richer features in subsequent training; at the same time, the edge detection map with continuous and complete visual content is used as a training sample, so that the adaptation module of subsequent training can learn the logical expression when restoring or reconstructing the video content. Among them, the resolution, frame rate and duration of each training video can be consistent with the target video to be generated.

[0074] S220, in each round of training, input the training video into the adaptation module, and save the fine-tuning vector output by the first hidden layer of each layer respectively.

[0075] In this step, the model architecture of the adaptation module is pre-set. Figure 2 As shown, a trained diffusion model with a corresponding network structure U-Net is selected, and the network structure of the adaptation module is set to be consistent with the network structure of the downsampling module in the diffusion model. In this embodiment, the number of layers of the first hidden layer of the adaptation module is consistent with the number of layers of the second hidden layer of the downsampling module of the diffusion model, and the vectors of the first hidden layer and the second hidden layer have the same dimension.

[0076] In some implementations, before training the adaptation module, the network parameters of the adaptation module are initialized. Specifically, the network parameters of the second hidden layers of each layer in the pre-trained diffusion model are used as the initialization parameters of the first hidden layers of each layer of the mapping of the adaptation module.

[0077] After initializing the adaptation module, the training of the adaptation module officially begins. After encoding and mapping each edge detection image of the training video to the latent space expression in the encoder, the latent space expression is input into the adaptation module, and the first hidden layer of each layer outputs the corresponding fine-tuning vector and saves it.

[0078] It should be noted that this step S220 can be performed synchronously and alternately with S230, that is, according to the number of layers of the first hidden layer, each time S220 is executed, S230 is executed once, S220 is executed again, and then S230 is executed once, until the last first hidden layer outputs a fine-tuning vector, that is, S220 is terminated. Alternatively, first according to S220, the fine-tuning vectors output by all the first hidden layers are saved in turn, and then S230 is executed.

[0079] S230, inputting random noise and training samples into the diffusion model to obtain a predicted noise image output by the diffusion model; wherein the output vector of each second hidden layer in the diffusion model is superimposed with the fine-tuning vector of the mapped first hidden layer as the input vector of the second hidden layer of the lower layer.

[0080] Optionally, the diffusion model and the adaptation module may share an encoder, or may use independent encoders. In this step, each frame of the edge detection image is randomly denoised and input into the diffusion model together with the text annotation. Referring to the operating principle of the diffusion model in step S120, the random noise is denoised for T time steps through the conditional guidance of the edge detection image to obtain a predicted noise image.

[0081] Taking the case where the number of layers of the first hidden layer is the same as that of the second hidden layer, in each time step, after the output vector of the second hidden layer of the first layer is obtained, it is added to the fine-tuning vector of the output of the first hidden layer of the first layer obtained in S220 to obtain the input vector of the second hidden layer of the second layer. After the output vector of the second hidden layer of the second layer is obtained, it is added to the fine-tuning vector of the output of the first hidden layer of the second layer to obtain the input vector of the second hidden layer of the third layer, and so on, until the output vector of the last second hidden layer is superimposed with the fine-tuning vector of the last first hidden layer to obtain the final output vector of the downsampling module, which enters the diffusion model through forward propagation for upsampling, and after T rounds of iterative denoising, it is input into the decoder, and finally the predicted noise image corresponding to each frame of the edge detection image is output.

[0082] S240, fixing the network parameters of each layer of the diffusion model, and adjusting the network parameters of the first hidden layer of each layer according to the back propagation of the loss function of the diffusion model to obtain a trained adaptation module.

[0083] In this application, the training process is to iteratively adjust the network parameters of the adaptation module, and the network parameters of each layer of the diffusion model are fixed. Based on this, after calculating the noise loss function of the true value pixels of the predicted noise image and the edge detection image, the network parameters of the first hidden layer of each layer of the adaptation module are adjusted by back propagation. In this way, according to the loop iteration from S220 to S240 until the loss function converges, a trained adaptation module can be obtained, and the first hidden layer of each layer has the trained network parameters.

[0084] From this example, it can be seen that the training method of the present application can be based on a small amount of high-quality sample videos, and the edge detection map of continuous expression of the content can be converted as a training video, and then an adaptation module can be connected to the pre-trained diffusion model to realize the training of the adaptation module, so as to efficiently and low-cost obtain a diffusion model that can restore or reconstruct the video content and is connected to the adaptation module.

[0085] See also Figure 4 According to the combination of the trained adaptation module and the pre-trained diffusion model in the above embodiment, the video generation method of the present application will be further introduced in combination with the embodiment below, which includes:

[0086] S310, according to a preset video frame rate and a preset duration, extract reference images of a corresponding number of frames in a single sample video in time sequence; and arrange the reference images in time sequence to form a video segment to be processed.

[0087] The user can select an existing sample video to generate a video clip to be processed. Optionally, the video frame rate of the sample video can be greater than or equal to the video frame rate of the video clip to be processed, and the duration of the sample video can be greater than or equal to the duration of the video clip to be processed. In order to obtain a video clip that meets the preset video frame rate and preset duration, multiple continuous reference frames that meet the preset duration and frame rate can be captured from the sample video.

[0088] If the original image in the sample video is composed of a line graph or an edge detection graph, the original image is directly used as the reference image required for the subsequent steps. If the original image in the sample video is not an edge detection graph, it can be converted into an edge detection graph through a related edge detection algorithm and then used as the input image for the subsequent steps S320 and S330.

[0089] S320, inputting the video clip to be processed into the pre-trained adaptation module to obtain the fine-tuning vector corresponding to the first hidden layer of each layer of the adaptation module; wherein the first hidden layer has the same number of layers and layer structure as the second hidden layer of the diffusion model.

[0090] This step is the same as the introduction of S120 and will not be repeated here.

[0091] S330, input the video clip and the prompt text into the pre-trained diffusion model to obtain the corresponding target video; wherein the output vector of the second hidden layer of each layer is used to be superimposed with the fine-tuning vector output by the mapped first hidden layer to serve as the input vector of the second hidden layer of the next layer.

[0092] In this embodiment, the condition information also includes a prompt text. The prompt text includes at least the attribute information of the main object in the target video, for example, it may include the type of the main object and other attribute information of the main object. The type of the main object may include: the type of person, the type of animal or other object (for example, an object that needs to be moved in a movie or game scene). Other attribute information of the main object may include the color information of the main object, the structure information (for example, when the main object is a person or an animal, it may include the body proportions of the main object) and other information.

[0093] As a general video generation model, the diffusion model has a relatively strong ability in text understanding. In this embodiment, by taking the prompt text as the input of the diffusion model, it can be ensured that the generated target video better meets the user's expectations.

[0094] Based on the expected goal, the prompt text is obtained based on the custom design and / or the visual content of the sample video. When the expected goal is to restore the video content, the prompt text needs to be as consistent as possible with the visual content of the sample video clip. When the expected goal is to reconstruct the video content, the prompt text can be different from the visual content of the sample video.

[0095] Referring to the description of S130 above, the fine-tuning vector output by each layer of S320 is superimposed with the output vector of the second hidden layer of the corresponding level to obtain a new vector with unchanged vector dimension but changed vector value, which is input into the second hidden layer of the next layer. Specifically, after random noise is added to each frame of the reference image in the video clip to be processed, it is input into the diffusion model together with the prompt text. According to the latent space expression corresponding to each frame of the encoded input image, each first hidden layer in the adaptation module outputs the corresponding fine-tuning vector Δd one by one; for example, the first hidden layer of the first layer outputs a fine-tuning vector Δd1, the first hidden layer of the second layer outputs a fine-tuning vector Δd2, and the first hidden layer of the nth layer outputs a fine-tuning vector Δd n .

[0096] Correspondingly, the output vector of the second hidden layer of the first layer in the diffusion model is d1. After d1 and Δd1 are superimposed, (d1+Δd1) is used as the input vector of the second hidden layer of the second layer. The output vector of the second hidden layer of the second layer is d2. After d2 and Δd2 are superimposed, (d2+Δd2) is used as the input vector of the second hidden layer of the third layer. Similarly, the output vector of the second hidden layer of the last layer is d n With Δd n After superposition, the final output vector is formed to be input into the upsampling module for upsampling. The downsampling module and the upsampling module perform denoising according to the prompt information of the prompt text to obtain the latent space expression that conforms to the visual content of the target video. After T rounds of iterative denoising, it is input into the decoder to obtain the target video.

[0097] The video generation method of the present application fine-tunes the denoising process of the diffusion model based on a pre-trained adaptation module, so that the diffusion model can obtain the denoising result affected by the adaptation module according to the prompt text, and realize the restoration or reconstruction of the content of multiple frames of continuous reference images, so as to combine the generated target images to form a target video to meet user needs.

[0098] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides a video generation device and corresponding embodiments.

[0099] Figure 5 It is a schematic diagram of the structure of the video generating device shown in this application.

[0100] See also Figure 5 The video generation device shown in the present application includes an image acquisition module 410, a model adaptation module 420 and a video generation module 430. Among them:

[0101] The image acquisition module 410 is used to acquire a video segment to be processed, where the video segment includes multiple frames of reference images.

[0102] The model adaptation module 420 is used to input the video clip to be processed into the pre-trained adaptation module to obtain the fine-tuning vector corresponding to the first hidden layer of each layer of the adaptation module; wherein the first hidden layer has the same layer structure as the second hidden layer of the diffusion model.

[0103] The video generation module 430 is used to input conditional information into the diffusion model to obtain the corresponding target video; wherein the output vector of each second hidden layer is used to be superimposed with the fine-tuning vector output by the mapped first hidden layer to serve as the input vector of the second hidden layer of the next layer.

[0104] See also Figure 6 In a specific implementation, the image acquisition module 430 is used to extract reference images of a corresponding number of frames in a single sample video in time sequence according to a preset video frame rate and a preset duration; and arrange the reference images in time sequence to form a video segment to be processed.

[0105] When the original image in the sample video is not an edge detection image, the apparatus of the present application further includes an edge detection module 440 for converting the original image of the corresponding number of frames in the sample video into an edge detection image.

[0106] From this example, it can be seen that the video generation device of the present application can insert an adaptation module into a pre-trained diffusion model, generate additional parameters through the adaptation module to adjust the downsampling process of the diffusion model, and adapt to the content production of new videos while retaining the general knowledge of the diffusion model, and restore or reconstruct the content of low-quality reference images in combination with prompt texts as much as possible.

[0107] An embodiment of the present application also shows a training device for an adaptation module, which includes:

[0108] The sample acquisition module is used to obtain multiple groups of training samples; each group of training samples includes a training video and corresponding text annotations, and the training video contains multiple frames of edge detection images continuously extracted from the same sample video.

[0109] The fine-tuning vector module is used to input the training video into the adaptation module in each round of training, and save the fine-tuning vector output by the first hidden layer of each layer.

[0110] The model training module is used to input random noise and training samples into the diffusion model to obtain the predicted noise image output by the diffusion model; wherein the output vector of each second hidden layer in the diffusion model is superimposed with the fine-tuning vector of the first hidden layer mapped as the input vector of the second hidden layer of the lower layer.

[0111] The parameter iteration module is used to fix the network parameters of each layer of the diffusion model, and adjust the network parameters of the first hidden layer of each layer according to the back propagation of the loss function of the diffusion model to obtain the trained adaptation module.

[0112] The adaptation module training device further includes an initialization module for using the network parameters of the second hidden layer of each layer of the pre-trained diffusion model as the initialization parameters of the first hidden layer of each layer before the fine-tuning vector module training.

[0113] The training device of the adaptation module of the present application adapts to specific downstream tasks by inserting a small, trainable adaptation module into a pre-trained diffusion model without fine-tuning the entire diffusion model, allowing the diffusion model to adapt to new tasks by adjusting a small number of parameters while keeping the original pre-trained parameters unchanged, thereby improving parameter efficiency and training speed.

[0114] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0115] Figure 7 It is a schematic diagram of the structure of the electronic device shown in this application.

[0116] See also Figure 7 , the electronic device 1000 includes a memory 1010 and a processor 1020 .

[0117] The processor 1020 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0118] The memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0119] The memory 1010 stores executable codes, and when the executable codes are processed by the processor 1020 , the processor 1020 can execute part or all of the methods described above.

[0120] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0121] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0122] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A video generation method, characterized in that: include: Acquire a video clip to be processed, wherein the video clip includes multiple frames of reference images; Input the video clip to be processed into a pre-trained adaptation module to obtain a fine-tuning vector corresponding to each first hidden layer of the adaptation module; wherein the first hidden layer has the same layer structure as the second hidden layer of the diffusion model; The conditional information is input into the diffusion model to obtain the corresponding target video; wherein the conditional information at least includes the video clip to be processed, and the output vector of each second hidden layer is used to be superimposed with the fine-tuning vector output by the mapped first hidden layer to serve as the input vector of the second hidden layer of the next layer.

2. The method according to claim 1, characterized in that The multiple frames of reference images are edge detection images or line drawings.

3. The method according to claim 1, characterized in that The step of obtaining the video clip to be processed includes: According to the preset video frame rate and preset duration, reference images of corresponding frames are extracted in time sequence from a single sample video; The reference images are arranged in time sequence to form the video segment to be processed.

4. The method according to claim 1, characterized in that: The condition information also includes prompt text; Before inputting the condition information into the diffusion model to obtain the corresponding target video, the method further includes: The prompt text is obtained based on a custom design and / or according to the visual content of a sample video for the intended purpose.

5. The method according to claim 1, characterized in that The first hidden layer has the same layer structure as the second hidden layer in the diffusion model, including: The number of layers of the first hidden layer of the adaptation module is equal to the number of layers of the second hidden layer and corresponds layer by layer; The vector dimension of the first hidden layer is the same as the vector dimension of the second hidden layer of the corresponding level.

6. The method according to claim 1, characterized in that Combined with the pre-trained diffusion model, the adaptation module is trained according to the following method: Acquire multiple groups of training samples; wherein each group of training samples includes a training video and corresponding text annotations, and the training video includes multiple frames of edge detection images continuously extracted from the same sample video; In each round of training, the training video is input into the adaptation module, and the fine-tuning vector output by the first hidden layer of each layer is saved respectively; Inputting random noise and the training sample into the diffusion model to obtain a predicted noise image output by the diffusion model; wherein the output vector of each second hidden layer in the diffusion model is superimposed with the fine-tuning vector of the first hidden layer mapped as the input vector of the second hidden layer of the lower layer; The network parameters of each layer of the diffusion model are fixed, and the network parameters of the first hidden layer of each layer are adjusted according to the back propagation of the loss function of the diffusion model to obtain a trained adaptation module.

7. The method according to claim 6, characterized in that The method further comprises: Before training the adaptation module, the network parameters of the second hidden layer of each layer in the pre-trained diffusion model are used as the initialization parameters of the first hidden layer of each layer mapped by the adaptation module.

8. A video generating device, characterized in that: include: An image acquisition module, used to acquire a video clip to be processed, wherein the video clip includes multiple frames of reference images; A model adaptation module, used for inputting the video clip to be processed into a pre-trained adaptation module to obtain a fine-tuning vector corresponding to each first hidden layer of the adaptation module; wherein the first hidden layer has the same layer structure as the second hidden layer of the diffusion model; A video generation module is used to input conditional information into the diffusion model to obtain a corresponding target video; wherein the conditional information at least includes the video clip to be processed, and the output vector of each second hidden layer is used to superimpose the fine-tuning vector output by the mapped first hidden layer to serve as the input vector of the second hidden layer of the next layer.

9. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the video generating method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the video generation method according to any one of claims 1 to 7.

Images