Video generation method and device, storage medium and program product

Abstract

The invention discloses a video generation method and device, a storage medium and a program product, and relates to the technical field of artificial intelligence, and the method comprises the steps: screening a target video frame from a current first video, and outputting a first feature vector through a feature injector; generating a second feature vector based on the preset noise and the description text of the current first video by using the text video model so as to generate a corresponding condition feature; determining an anchor frame of the current first video, generating a target vector according to the anchor frame and the description text, and further generating a fusion feature; and generating a new current first video based on the fusion feature, skipping to the step of screening the target video frame, and generating a corresponding second video until the sum of the frame number of each first video meets a preset condition. By capturing the visual features of the historical frames and determining the fusion features according to the anchor frames and the description texts, forgetting can be prevented in the future video frame generation process, the problems of feature forgetting, discontinuous time sequence and the like are solved, and the consistency and smoothness of the generated video are kept.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video generation method, device, storage medium, and program product. Background Art

[0002] The AIGC (Artificial Intelligence Generated Content) movement is rapidly gaining momentum, and OpenAI recently released SoRA, a large-scale model for video generation. The SoRA model can generate high-definition videos from text descriptions, offering a new approach for rapid prototyping and proof-of-concept in creative content production, film, animation, gaming, and advertising.

[0003] While current video models can generate short videos that follow textual content, they primarily use a cascaded approach to continuously generate new videos. Consequently, because the currently generated video frame is used as input to predict future frames, this often leads to forgetting over time. Generating new video frames not only results in spatial content errors (e.g., changes in the dog's facial features and color in the new video frame), but also in discontinuities and dramatic jumps in timing. Therefore, generating high-quality long videos is a worthy research topic. Summary of the Invention

[0004] The present application provides a video generation method, device, storage medium and program product to at least solve the problem in the related art that the prediction of future frames of a video generates content errors and temporal discontinuities as time increases.

[0005] This application provides a video generation method, comprising:

[0006] Filtering a preset number of target video frames from the current first video, and outputting a plurality of first feature vectors corresponding to the target video frames using a preset feature injector;

[0007] Generate a plurality of second feature vectors based on the preset noise and the description text corresponding to the current first video using a preset Wensheng video model, and generate corresponding conditional features based on the first feature vector and the second feature vector;

[0008] Determine an anchor frame in the current first video, generate a corresponding target vector according to the anchor frame and the description text, and determine a corresponding fusion feature based on the target vector, the first feature vector, and the conditional feature;

[0009] Generate a new current first video based on the fused features, and jump to the step of screening a preset number of target video frames from the current first video. Until the sum of the number of frames of each first video meets the preset frame number condition, generate corresponding second videos based on each first video.

[0010] This application also provides an electronic device, including: a memory for storing a computer program; a processor for implementing the steps of any of the above video generation methods when executing the computer program.

[0011] This application also provides a computer-readable storage medium, in which a computer program is stored. Wherein, the computer program implements the steps of any of the above video generation methods when executed by a processor.

[0012] This application also provides a computer program product, including a computer program, which implements the steps of any of the above video generation methods when executed by a processor.

[0013] Through this application, the features of the target video frames can be output by the feature injector, so as to capture the visual features of historical frames, maintain the continuity of future generated video frames, and the anchor frames in the current video can be determined, and corresponding target vectors can be generated according to the anchor frames and the description text, and the fused features can be determined, so that the high-level scene semantics can be captured from the earlier history and the forgetting can be prevented during the generation process of future video frames. Since the currently generated video frames are used as the next input for predicting future video frames, the problems of feature forgetting and temporal discontinuity can be solved, and the consistency and smoothness of generating long videos can be maintained. Description of the Drawings

[0014] To illustrate the embodiments of the present application more clearly, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0015] Figure 1 It is a flowchart of a video generation method provided by an embodiment of the present application;

[0016] Figure 2 It is a framework diagram of a regression-based long video generation network provided by an embodiment of the present application;

[0017] Figure 3 It is a framework diagram of a regression-based long video generation network provided by an embodiment of the present application;

[0018] Figure 4 It is a network structure diagram of an attention model provided by an embodiment of the present application;

[0019] Figure 5 A schematic diagram of long video segmentation provided by an embodiment of the present application;

[0020] Figure 6 A schematic diagram of a key frame generation diffusion model network provided by an embodiment of the present application;

[0021] Figure 7 A schematic diagram of a cross-attention sampling layer provided by an embodiment of the present application;

[0022] Figure 8 A schematic diagram of a key frame interpolation diffusion model network provided by an embodiment of the present application;

[0023] Figure 9 A flowchart of short video content inference generation provided by an embodiment of the present application;

[0024] Figure 10 A schematic diagram of a key frame extraction network model provided by an embodiment of the present application. Detailed implementation manners

[0025] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0026] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects and not to describe a specific order or sequence.

[0027] Currently, the text-to-video model mainly generates new videos by cascading short videos. Specifically, the currently generated video frames are used as the next input for predicting future video frames. However, as time increases, problems such as spatial content errors and temporal discontinuities usually occur. The present application can use the feature injector to output the features of the target video frame, thereby capturing the visual features of historical frames and maintaining the continuity of future generated video frames. Moreover, the anchor frame in the current first video can be determined, and after generating the corresponding target vector based on the anchor frame and the description text, the fusion feature can be determined, so as to prevent forgetting during the generation process of future video frames and maintain the consistency and smoothness of the generated long video.

[0028] To enable those skilled in the art of the present technology to better understand the solution of this application, the following further detailed description of this application will be given in conjunction with the accompanying drawings and specific embodiments.

[0029] It should be noted that this application is mainly applied to the text-to-video scenario of AIGC. Among them, it can be understood that the input of the text-to-video task is a text description, and the output is a video that matches the content of the text description. Affected by computing power, the current mainstream text-to-video models can usually only generate short videos within 30 frames. This application proposes a fast long-video content generation system architecture. By only using a traditional text-to-image model and adopting a regression-based generation method, a long-video generation model (more than 1000 frames) can be generated. Through the regression-based generation method of the generation content system architecture in this application, a parallel hierarchical parallel generation method can be adopted, which greatly improves the generation efficiency.

[0030] Specifically, an embodiment of this application provides a video generation method, including:

[0031] Step S11: Screen a preset number of target video frames from the current first video, and use a preset feature injector to output a number of first feature vectors corresponding to the target video frames.

[0032] In this embodiment, as Figure 2 shown, it is necessary to screen a preset number of target video frames from the current first video. Specifically, first, based on a preset description text, for example: "A person is skating on an ice-covered island full of trees", and then use a pre-trained key-frame text-to-image model and key-frame interpolation model to generate a corresponding video segment, that is, the current first video. For example, use the key-frame text-to-image model and the key-frame interpolation model to infer a video segment consistent with the input text. However, it should be noted that the video segment at this time is usually only about 2s. Adopting the regression-based long-video generation network framework as Figure 2 shown, long-video content (a video of 1 minute or even longer) can be generated recursively starting from the initial content video frame (32 frames). It should be noted that in this embodiment, a historical frame conditional attention module, a long-term memory perception module, and a random enhancement hybrid inference module are added to the traditional text-to-video model (such as the ModelScope model). Among them, the network modules that need to be trained are the historical frame conditional attention module, the MLP (Multilayer Perceptron) module and the convolutional module in the long-term memory perception module. Therefore, the training parameters account for a very small proportion compared with the text-to-video model, resulting in extremely low computing power consumption, but high-quality long-video content can be generated.

[0033] Among them, the historical frame conditional attention module injects the tail content of the previously generated video content into the text-to-video model as a condition, enabling it to generate videos with temporal content consistency when generating new video content; the long-term memory perception module ensures spatial content consistency during the generation of new video content and does not lose the details of objects in the video; the random enhancement hybrid inference module does not require training any network parameters and can obtain high-quality long videos through a hybrid sampling strategy combined with a high-quality text-to-video model.

[0034] Specifically, after obtaining the current first video content V (video length 32 frames) through the key frame text-to-image model and the key frame interpolation model, a number of first feature vectors corresponding to the target video frame are output using a preset feature injector. Specifically, the target video frame can be processed using a preset image encoder and convolutional layers, and Gaussian noise is added to the processed target video frame to obtain a noise-added target video frame, which is then input into the preset feature injector to output the first feature vectors corresponding to the target video frame at several levels. Among them, the above levels are divided based on the network structure of the preset image encoder, and the network structure and initialization parameters of the preset image encoder are the same as those of the preset feature injector.

[0035] Such as Figure 2 shown, the current first video in this embodiment can be the video content output by the key frame interpolation model or the video content generated by the long video generation framework in the previous time in this embodiment, and then the last 16 frame images in the video content are intercepted , and the initial video condition vectors at each stage are obtained through the historical frame conditional attention module. It should be noted that, as Figure 2 shown, the historical frame conditional attention module in this embodiment consists of an image encoder and a feature injector. Among them, the above image encoder adopts the network structure of ViT-L (Vision Transformer Low resolution, a low-resolution variant of the ViT model) (the specific type is not limited).

[0036] Step S12: Use the preset text-to-video model to generate a number of second feature vectors based on the preset noise and the description text corresponding to the current first video, and generate corresponding conditional features based on the first feature vectors and the second feature vectors.

[0037] In this embodiment, the preset text-to-video model can be used to generate a number of second feature vectors based on the preset noise and the description text corresponding to the current first video , and based on the first feature vectors and the second feature vectors generate corresponding conditional features . Specifically, After the image encoder and zero convolution (the convolution layer is randomly initialized with a parameter mean of 0) and Gaussian noise is added, and then the feature vectors of each stage are obtained through the feature injector. (i represents the index of the network stage, such as Figure 2 As shown in the figure, there are 4 stages). It can be understood that the feature injector network structure and Figure 2 The encoder of the Chinese video model is the same, and the initialization parameters also load the network weights of the encoder of the Chinese video model. The feature vectors of each stage obtained by the Vincent video model encoder Input to the attention model.

[0038] That is to say, when generating corresponding conditional features based on the first eigenvector and the second eigenvector in this embodiment, the second eigenvector can be normalized using the preset spatiotemporal normalization layer, and the normalized second eigenvector can be processed using the first preset fully connected layer, and then the first eigenvector and the processed second eigenvector are fused through the preset temporal attention layer, and the fused eigenvector is processed using the second preset fully connected layer to obtain the conditional features. Specifically, the network structure of the above attention model is as follows: Figure 3 As shown, After the spatiotemporal normalization layer Norm and the fully connected layer ,and Pass through the temporal attention layer together , then through the fully connected layer It is worth noting that the above temporal attention layer includes 、 、 The parameters of the three fully connected layers are as follows: is the dimension of the key vector, used to scale the dot product:

[0039] ;

[0040] Accordingly, Figure 3 The specific formula of the attention model network structure diagram shown is as follows:

[0041] ;

[0042] ;

[0043] ;

[0044] .

[0045] Step S13: Determine the anchor frame in the current first video, generate a corresponding target vector based on the anchor frame and the description text, and determine the corresponding fusion feature based on the target vector, the first feature vector, and the conditional feature.

[0046] In this embodiment, the anchor frame in the current first video can be determined first, a corresponding target vector can be generated based on the anchor frame and the description text, and the corresponding fusion feature can be determined based on the target vector, the first feature vector, and the conditional feature. Specifically, when generating a corresponding target vector based on the anchor frame and the description text, the text encoder of the preset multi-modal training model can be used to encode the description text to obtain the corresponding text feature vector, and the image encoder of the preset multi-modal training model can be used to encode the anchor frame to obtain the corresponding image feature vector. Then, a preset multi-layer perceptron is used to perform feature mapping on the image feature vector according to the vector dimension of the text feature vector, and the text feature vector and the mapped image feature vector are concatenated to obtain a concatenated feature vector. After that, the weights corresponding to each level of the preset text-to-video model are determined, and a corresponding target vector is generated based on the weights and the concatenated feature vector.

[0047] Specifically, as Figure 2 shown, in this embodiment, anchor frame c and the corresponding text description are first selected. For example, a frame can be randomly selected from as the anchor frame and input into the CLIP model (Contrastive Language-Image Pre-training, a multi-modal pre-training model). For example, the description text vector obtained by the CLIP text encoder is , with a dimension of , where b is the batch size; the vector obtained by the image encoder is which is . After that, it goes through an MLP (multi-layer perceptron) for feature mapping, mapped to the same dimension as , and then concatenated with in the last dimension to obtain a vector with a dimension of . Then, through one-dimensional convolution and normalization, the feature is mapped to , denoted as . It should be noted that a weight needs to be initialized for each stage in the network here (the number of stages here is different from that of the text-to-video model and the feature injector. As Figure 2 shown, it is 8), as shown in the figure . Then and can pass through the modal content interaction layer to obtain the target vector, that is, the modal interaction vector :

[0048] ;

[0049] After combining with the input random noise, the vectors generated by each stage of the text-to-video model (such as ModelScope), the fused features can be calculated :

[0050] ;

[0051] Among them, as shown in the above formula, in this embodiment, the last four features are taken.

[0052] Step S14: Generate a new current first video based on the fused features, and jump to the step of screening a preset number of target video frames from the current first video until the sum of the number of frames of each first video meets the preset frame number condition, and then generate the corresponding second video based on each first video.

[0053] In this embodiment, a new current first video is generated based on the fused features, and then as Figure 2 shown, jump to the step of screening a preset number of target video frames from the current first video until the sum of the number of frames of each first video meets the preset frame number condition, and then generate the corresponding second video based on each first video. That is to say, After one forward pass through the text-to-video model for one denoising, repeat the above steps of selecting target video frames to denoising the fused features for a total of T times (T can take an empirical value, not specifically limited. For example, in this embodiment, T = 5) to complete T times of denoising, and finally obtain the next video content through the decoder, with a length of 32 frames. Then, use the obtained video content as the new initial video content again, and loop the steps of the previous new video until the length of the video content that meets the user's requirements is generated to obtain the corresponding long video. That is to say, in the process of generating a new current first video based on the fused features in this embodiment, as Figure 2 shown, the random enhancement hybrid inference module can be used to improve the spatial resolution of the long video content generated in the previous step to complete the generation of high-quality video content. Specifically, the current fused features can be input into a preset image encoder, and the feature vector output by the preset image encoder based on the current fused features can be used as the new first feature vector, and then jump to the step of generating the corresponding conditional features based on the first feature vector and the second feature vector until the current fused features meet the preset denoising condition, and then generate the new current first video based on the current fused features. It can be understood that the above preset denoising condition can be the number of times of re-inputting the fused features into the preset image encoder.

[0054] It should be noted that the first video in this embodiment is a short video whose video duration meets the preset short-duration determination condition, and the second video is a long video whose video duration meets the preset long-duration determination condition. That is to say, in this embodiment, both the first video and the second video are videos that meet the corresponding duration or frame number conditions, and the above conditions can be adjusted according to the actual situation of the model, and are not specifically limited to videos of a certain duration. It can be understood that since the second video is generated from multiple first videos, the duration or frame number of the second video is greater than that of the first video. For example, the first video in this embodiment can be a video with 32 frames, while the second video is a video with more than 1000 frames or a video with a duration greater than 1 minute.

[0055] Specifically, the second video can be segmented based on a preset video segmentation length to obtain a number of segmented videos; among them, there are target number of frame repeated video frames between adjacent segmented videos; then Gaussian noise random sampling is performed on each segmented video to obtain a sampled video, and the preset video diffusion super-resolution model is used to denoise the sampled video according to the second feature vector, and an optimized second video is obtained based on the denoised sampled video. During the process of denoising the sampled video, the current sampled video can be denoised first to obtain a first hidden vector , and the next sampled video of the current sampled video is denoised to obtain a second hidden vector , then the repeated video frames between the current sampled video and the next sampled video are determined, and the third hidden vector and the fourth hidden vector corresponding to the repeated video frames in the first hidden vector and the second hidden vector are determined respectively, and then the current sampled video is denoised based on the third hidden vector and the fourth hidden vector.

[0056] Specifically, as Figure 4 shown, first, the second video can be evenly segmented, each segment contains L frames (32 frames in this embodiment), and there are M frames of repetition between adjacent segments (8 frames in this embodiment as Figure 4 shown), and then Gaussian noise random sampling is performed on each video segment:

[0057] ;

[0058] ;

[0059] Then is combined with Fi and the video diffusion super-resolution model (such as MS-Vid2Vid-XL) is used for denoising, the denoising step t = 100 steps, and the diffusion step of the whole process is T = 1000 steps, and then the denoised hidden vector It can be understood that the reason for t = 100 steps here is that in the previous steps of the denoising process, the overall content structure of the video is mainly maintained, while in the subsequent denoising process, the details of the video content are gradually increased.

[0060] As Figure 4 shown, after that, the random mixing strategy is executed. After performing t-step denoising operations on the (i - 1)-th video segment and the i-th video segment respectively, the denoised latent vectors and are obtained, and the denoised latent vectors corresponding to the corresponding parts of the common covered video frames are selected and . Among them, covers the smooth time-domain content information from the starting frame of the -th video segment to the common covered video frame; covers the smooth time-domain content information from the common covered video frame of the -th video segment to its ending frame.

[0061] When denoising the current sampled video, specifically, the random video frame position can be determined from the repeated video frames, and according to the random video frame position, the corresponding first video frame in the current sampled video can be determined. Then, according to the next video frame position of the random video frame position, the corresponding second video frame in the next sampled video can be determined. After that, the first vector to be stitched corresponding to the first video frame is determined from the third latent vector, and the second vector to be stitched corresponding to the second video frame is determined from the fourth latent vector, and the first vector to be stitched and the second vector to be stitched are stitched to obtain the stitched vector. The current sampled video is denoised according to the stitched vector, and then the next sampled video is used as the new current sampled video, and it jumps to the step of denoising the current sampled video until the denoising of each sampled video is completed.

[0062] That is to say, in order to solve the inconsistency of the generated content between the two video segments, in this embodiment, the denoised latent vectors and can be combined. For example, a number m is randomly selected from [0, M], and then two adjacent frame latent vectors starting from this number are selected from and (for example, selects the m-th latent vector, selects the (m + 1)-th latent vector), and the vector stitching operation is performed. In this way, the denoised latent vectors between the segments can interact with each other, reducing the content inconsistency generated by the super-resolution model in the video sub-segments. Then, the next denoising of the video segment is performed. For the denoising of the subsequent frames of the video segment , the corresponding denoised latent vector needs to be multiplied by the denoising probability, and the denoising probability is specifically , by performing 9,000 denoising operations, the super-resolution denoising process of the th segment is completed. By performing the above denoising process on all video segments accordingly, the super-resolution denoising process of the entire long video can be completed. In this way, through the random enhancement hybrid inference module, the spatial resolution of the generated long video content can be improved, and the generation of high-quality video content can be completed.

[0063] In this embodiment, aiming at the problem that the current text-to-video model cannot generate long videos with high quality, a regression-based long video generation system is designed. The long video content is generated through an autoregressive network. Among them, the autoregressive network module mainly includes a historical frame conditional attention network module, a long-term memory perception module, and an autoregressive video random enhancement hybrid module. The historical frame conditional attention network module can capture the visual features of historical frames and maintain the continuity of future generated video frames; the long-term memory perception module can capture high-level scene semantics from earlier history and prevent forgetting during the generation process of future video frames; the autoregressive video random enhancement hybrid module processes video segments with visually repetitive content and maintains the consistency and smoothness of the generated long video.

[0064] Based on the previous embodiment, it can be known that this application can use the feature injector to output the features of the target video frame, generate the corresponding target vector according to the anchor frame and the description text, and then determine the fusion feature to generate the future video frame. It should be noted that when the current large model using longer visual tokens generates a long video model, such a large model not only requires more data, but also requires a very large computing power cluster. Therefore, limited by computing power, it is difficult to generate a long video model with higher quality. Therefore, aiming at the problem that the current text-to-video model cannot generate long videos with high quality with low computing power requirements, a regression-based long video generation system is designed in combination with the previous embodiment. Next, the process of obtaining the initial target video frame in this embodiment will be described in detail. Refer to Figure 5 As shown, the embodiment of this application provides a specific video frame acquisition method, including:

[0065] Step S21, use a preset key frame generation model to generate an initial key frame corresponding to the description text.

[0066] In this embodiment, it can be understood that generally, to go from a text-to-image model to a text-to-video model, a temporal attention module needs to be designed on the original basis. However, the additional temporal attention module needs to be trained with a large number of specific video-text pair datasets to obtain a better text-to-video model, and video-text pairs consume much more computing power than image-text pairs. The short video generation model proposed in this solution designs a new regression-based generation method. On the one hand, only a traditional text-to-image network model is needed, without training the temporal attention module, greatly reducing the computing power loss. On the other hand, with the new regression-based generation method, the video frames can be output in a hierarchical and parallel manner, greatly improving the generation efficiency.

[0067] It should be noted that the short video generation network model in this embodiment mainly includes a key-frame text-to-image model (i.e., the key-frame generation model) and a key-frame interpolation model, both of which adopt a generation method based on grid pictures. The input of the key-frame text-to-image model is Gaussian noise of 512*512 and text input, and the output is four key-frame pictures, each picture being 256*256. The temporal order of the key frames is from left to right and from top to bottom. Key-frame interpolation can be regarded as an image editing problem. Place the key-frame pictures in the upper left and lower right corners, and Gaussian noise in the upper right and lower left corners. The picture size is also 256*256, and then generate the video frames between the two key frames based on the text input. It should also be noted that for crawling video data from the network, first, a key-frame selection model needs to be made. The criterion for key-frame selection is scene change and motion change. Correspondingly, in this embodiment, the key frames of the video corresponding to the text description can be extracted based on the key-frame selection network of the large model. For each video V, the corresponding video description C and key-frame sequence S are obtained. After obtaining the corresponding data, the model training for key-frame generation and the model training for key-frame interpolation can be carried out based on the existing text-to-image model.

[0068] Specifically, in this embodiment, first, the initial key frames corresponding to the description text can be generated using the preset key-frame generation model. Specifically, the corresponding key frames can be extracted from the preset video using the preset key-frame extraction model, and the key frames are spliced to obtain the corresponding mixed image. Then, the Markov chain hidden vector is generated using the initial key-frame generation model, and the Markov chain hidden vector is added to the mixed image. After that, the cross-attention layer in the initial key-frame generation model is used to denoise the mixed image based on the Markov chain hidden vector. After obtaining the denoised image, the initial key-frame generation model is trained based on the denoised image and the mixed image until the current loss value of the initial key-frame generation model meets the first preset training condition, and the current initial key-frame generation model is used as the preset key-frame generation model.

[0069] Such as Figure 6As shown, the key frame generation model is a two-dimensional convolutional diffusion model. In the key frame generation diffusion model network, the model adopts an encoder-decoder architecture, mainly composed of an encoder-decoder, a cross-attention downsampling layer, and a cross-attention upsampling layer. Among them, the network structure of each cross-attention layer is as Figure 7 shown, where the downsampling is a two-fold downsampling of the feature map, and the upsampling is a two-fold upsampling of the feature. Moreover, the network structures of the upsampling network and the downsampling network are the same, mainly composed of a residual module, a self-attention mechanism module, a feed-forward network module, and a cross-attention module. Among them, the residual module adopts the basic network module of the residual network ResNet.

[0070] When training the initial key frame generation model, first, a mixed image composed of four key frames spliced and combined in the order from left to right and from top to bottom in the way is given . Then, the diffusion model first generates a series of Markov chain latent vectors . This series of latent vectors gradually add Gaussian noise to the original image . The formula for adding Gaussian noise is as follows:

[0071] ;

[0072] The specific formula for the denoising process based on the latent vector is as follows:

[0073] ;

[0074] Specifically, during the image generation process, the initial is randomly sampled from the Gaussian noise , and the noise is gradually reduced to obtain the real image . It can be understood that in the specific practice process, usually the deviation of the predicted noise rather than the real image is used as the specific optimization training function. The specific function is as follows:

[0075] ;

[0076] where is randomly sampled from the Gaussian noise , is the deviation obtained by the neural network inference of the input time step t and the noisy image at time step t under the input condition c, is the parameter learned by the neural network, and c represents various condition inputs, which refers to the text description in this embodiment.

[0077] Then, a loss function is used for training judgment. The specific loss function used is a loss function that combines deviation prediction and actual value prediction as a regularization term. The overall loss function is as follows:

[0078] ;

[0079] At this time, represents the number of currently learnable parameters, and the latter term as a regularization term can reduce the risk of overfitting.

[0080] Step S22: Use a preset key-frame interpolation model to perform interpolation based on the initial key frames to obtain the current first video; the preset key-frame generation model and the preset key-frame interpolation model are models constructed based on a two-dimensional convolutional diffusion model.

[0081] In this embodiment, a preset key-frame interpolation model can be used to perform interpolation based on the initial key frames to obtain the current first video. Among them, as in the previous step, both the preset key-frame generation model and the preset key-frame interpolation model are models constructed based on a two-dimensional convolutional diffusion model. The training process of the key-frame interpolation model specifically includes: First, construct an initial key-frame interpolation model based on the network architecture and model weights of the preset key-frame generation model, generate corresponding Gaussian noise according to the resolution of the key frames, construct an initial input image based on the key frames and the Gaussian noise, and then input the initial input image into the initial key-frame interpolation model to perform interpolation between the key frames to obtain corresponding inserted video frames. After that, train the initial key-frame interpolation model based on the inserted video frames and the key frames until the current loss value of the initial key-frame interpolation model meets the second preset training condition, and use the current initial key-frame interpolation model as the preset key-frame interpolation model.

[0082] Specifically, as Figure = 8 shown, the key-frame interpolation model changes the input of the key-frame generation model, and the overall network architecture is the same. Therefore, the key-frame generation model can be loaded with the initial weights of the network model, and the training method and loss function are the same as those of the key-frame generation model. Correspondingly, the process of the key-frame text-to-image model and the interpolation model inferring the initial video frames is as Figure 9 shown: For the user input text, first generate key frames related to the text input based on scene transformation and motion transformation according to the key-frame text-to-image model, for example Figure 9The images of the 1st frame, 10th frame, 21st frame, and 30th frame are generated. Then, for adjacent key frames, the key-frame interpolation model is used to perform interpolation in a recursive parallel manner. For example, the adjacent key frames are placed in the upper left corner and the lower right corner, with resolutions of 256*256 respectively. Then, Gaussian noise is placed in the lower left corner and the upper right corner respectively, thus forming an initial input of 512*512. Then, the initial input obtained by mixing the key frames and Gaussian noise is input into the key-frame interpolation model for interpolation to generate intermediate video frames. For example, the 1st frame and the 10th frame generate intermediate frames of the 4th frame and the 7th frame, and the 21st frame and the 30th frame generate intermediate frames of the 24th frame and the 27th frame. And it can be understood that if a high-frame-rate video continues to be generated, the key-frame interpolation model can continue to be recursively and parallelly called.

[0083] Before using the preset key-frame extraction model to extract corresponding key frames from the preset video, a training video can also be obtained, corresponding video content text can be generated according to the training video, and corresponding positive and negative samples can be constructed based on the training video and the video content text, so as to construct a training set according to the positive and negative samples. Then, the image encoder in the initial key-frame extraction model is used to convert the training videos in the training set into corresponding video frame vectors, and the key-frame probabilities corresponding to each video frame of the training video are determined according to the video frame vectors, so as to train the initial key-frame extraction model based on the key-frame probabilities and the training video until the current loss value of the initial key-frame extraction model meets the third preset training condition, and the current initial key-frame extraction model is used as the preset key-frame extraction model.

[0084] That is to say, as Figure 10 shown, when training the video key-frame extraction model, first obtain the Internet video set V (the length of the V video is limited within 10s), and then input each video into the Video Caption model (such as the GPT4V API) to obtain a detailed video content description , and obtain the data pair . After that, positive and negative samples are constructed for the data pair . The positive sample is . There are the following ways for negative samples: The first is to randomly replace the objects or attributes in to obtain ; The second is to retrieve descriptions similar to from the set to form . Then, control the positive and negative sample ratio to 1:2 to obtain the training set D, and according to Figure 10 .The network model shown uses D for the training of the key frame extraction network. Specifically, in this embodiment, the image encoder uses ViT, and the key frame network (with a network structure of Bert-Base and also loaded with the initial weights of Bert-Base) is the network weights to be trained. The video frames pass through the image encoder to obtain frame vectors . Among them, the query vector is the randomly initialized parameter q. q and the frame vectors are input into the key frame network to obtain the solidified vector h. It should be noted that the role of q is to fix it to the same time dimension regardless of the number of n. For example, the time dimension in this embodiment is 100. Then, using h, combined with the question prompt and the video description, input into the large language model (such as vicuna-13b), the large language model extracts the key frames that match this video description, and then inputs the key frame features into the answer network, where the answer network and the key frame network share weights. In this way, the probability that the large language model outputs yes can be used as the probability that the video frame is a key frame, and the training loss is the cross-entropy loss used for the large language model to predict the next token task. After completing the training of the key frame network, for inference, the key frames of each video can be obtained .

[0085] Step S23: Use the last frame of the current first video as the starting frame, and sequentially screen a preset number of video frames in the current first video based on the starting frame to obtain the target video frames.

[0086] In this embodiment, the last frame of the current first video is used as the starting frame, and a preset number of video frames in the current first video are sequentially screened based on the starting frame, and the target video frames can be obtained. For example, in this embodiment, the last 16 frames of the current first video with 32 frames can be extracted as the target video frames.

[0087] Through the above technical solutions, in this embodiment, aiming at the problem that the current text-to-video model cannot generate high-quality long videos with low computing power requirements, a fast video content generation model based on the text-to-image model to generate short video sequences is designed, and the traditional two-dimensional convolutional diffusion model UNet is used instead of the three-dimensional convolutional diffusion model with higher computing power consumption. At the same time, the model does not need to train the temporal network module and adopts a regression-based space-for-time inference method, which can perform parallel computing and achieve efficient and fast inference of short video content.

[0088] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0089] An embodiment of the present application further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above video generation method embodiments.

[0090] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps in any of the above video generation method embodiments when running.

[0091] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media such as USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), mobile hard disks, magnetic disks, or optical discs that can store computer programs.

[0092] An embodiment of the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the steps in any of the above video generation method embodiments are implemented.

[0093] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above video generation method embodiments are implemented.

[0094] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0095] The above has introduced in detail a video generation method, device, storage medium, and program product provided by the present application. Specific examples are used herein to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A video generation method, characterized in that, Including: Screening a preset number of target video frames from the current first video, and outputting a plurality of first feature vectors corresponding to the target video frames by using a preset feature injector; Generating a plurality of second feature vectors by using a preset text-to-video model based on preset noise and the description text corresponding to the current first video, and generating corresponding conditional features based on the first feature vectors and the second feature vectors; Determining an anchor frame in the current first video, generating a corresponding target vector according to the anchor frame and the description text, and determining a corresponding fusion feature based on the target vector, the first feature vectors and the conditional features; Generating a new current first video based on the fusion feature, and jumping to the step of screening a preset number of target video frames from the current first video. After the sum of the number of frames of each first video meets a preset frame number condition, generating a corresponding second video based on each first video.

2. The video generation method according to claim 1, wherein The screening of a preset number of target video frames from the current first video includes: Generating an initial key frame corresponding to the description text by using a preset key frame generation model, and performing frame interpolation based on the initial key frame by using a preset key frame interpolation model to obtain the current first video; the preset key frame generation model and the preset key frame interpolation model are models constructed based on a two-dimensional convolutional diffusion model; Taking the last frame of the current first video as the starting frame, and sequentially screening the preset number of video frames in the current first video based on the starting frame to obtain the target video frames.

3. The video generation method according to claim 2, wherein Before screening a preset number of target video frames from the current first video, it further includes: Extracting corresponding key frames from a preset video by using a preset key frame extraction model, and splicing the key frames to obtain a corresponding mixed image; Generating a Markov chain hidden vector by using an initial key frame generation model, and adding the Markov chain hidden vector to the mixed image; Denosing the mixed image based on the Markov chain hidden vector by using a cross-attention layer in the initial key frame generation model to obtain a denoised image; Training the initial key frame generation model based on the denoised image and the mixed image until the current loss value of the initial key frame generation model meets a first preset training condition, and taking the current initial key frame generation model as the preset key frame generation model.

4. The video generation method according to claim 3, wherein Before screening a preset number of target video frames from the current first video, it further includes: Constructing an initial key frame interpolation model based on the network architecture and model weights of the preset key frame generation model; Generating corresponding Gaussian noise according to the resolution of the key frame, and constructing an initial input image based on the key frame and the Gaussian noise; Inputting the initial input image into the initial key frame interpolation model to perform frame interpolation between the key frames to obtain corresponding inserted video frames; Training the initial key-frame interpolation model based on the inserted video frames and the key frames until the current loss value of the initial key-frame interpolation model meets the second preset training condition, and using the current initial key-frame interpolation model as the preset key-frame interpolation model.

5. The video generation method according to claim 4, wherein Before using the preset key-frame extraction model to extract corresponding key frames from the preset video, it further includes: Obtaining a training video and generating corresponding video content text according to the training video; Constructing corresponding positive and negative samples based on the training video and the video content text, and constructing a training set according to the positive and negative samples; Using the image encoder in the initial key-frame extraction model to convert the training video in the training set into corresponding video frame vectors; Determining the key-frame probabilities corresponding to each video frame of the training video according to the video frame vectors; Training the initial key-frame extraction model based on the key-frame probabilities and the training video until the current loss value of the initial key-frame extraction model meets the third preset training condition, and using the current initial key-frame extraction model as the preset key-frame extraction model.

6. The video generation method according to claim 1, wherein The step of using the preset feature injector to output several first feature vectors corresponding to the target video frame includes: Processing the target video frame using a preset image encoder and a convolutional layer, and adding Gaussian noise to the processed target video frame to obtain a noisy target video frame; Inputting the noisy target video frame into the preset feature injector to output several levels of the first feature vectors corresponding to the target video frame; the levels are divided based on the network structure of the preset image encoder, and the network structure and initialization parameters of the preset image encoder are the same as those of the preset feature injector.

7. The video generation method according to claim 6, wherein The step of generating corresponding conditional features based on the first feature vector and the second feature vector includes: Normalizing the second feature vector using a preset spatio-temporal normalization layer and processing the normalized second feature vector using a first preset fully connected layer; Fusing the first feature vector and the processed second feature vector through a preset temporal attention layer and processing the fused feature vector using a second preset fully connected layer to obtain the conditional features.

8. The video generation method according to claim 7, wherein In the process of generating a new current first video based on the fused features, it further includes: Inputting the current fused features into the preset image encoder; Using the feature vector output by the preset image encoder based on the current fused features as a new first feature vector, and jumping to the step of generating corresponding conditional features based on the first feature vector and the second feature vector until the current fused features meet the preset denoising condition, and then generating a new current first video based on the current fused features.

9. The video generation method according to claim 1, wherein The step of generating a corresponding target vector according to the anchor frame and the description text includes: Encoding the description text using the text encoder of the preset multi-modal training model to obtain a corresponding text feature vector; Encode the anchor frame using the image encoder of the preset multi-modal training model to obtain a corresponding image feature vector; Using a preset multi-layer perceptron, perform feature mapping on the image feature vector according to the vector dimension of the text feature vector, and splice the text feature vector and the mapped image feature vector to obtain a spliced feature vector; Determine the weights corresponding to each level of the preset text-to-video model, and generate the corresponding target vector according to the weights and the spliced feature vector.

10. The video generation method according to any one of claims 1 to 9, characterized in that, After generating the corresponding second video based on each first video, it further includes: Segment the second video based on a preset video segmentation length to obtain a number of segmented videos; wherein, there is a target number of repeated video frames between adjacent segmented videos; Perform Gaussian noise random sampling on each of the segmented videos to obtain sampled videos; Use a preset video diffusion super-resolution model to denoise the sampled videos according to the second feature vector, and obtain an optimized second video based on the denoised sampled videos.

11. The video generation method according to claim 10, wherein The denoising of the sampled videos includes: Denoise the current sampled video to obtain a first hidden vector, and denoise the next sampled video of the current sampled video to obtain a second hidden vector; Determine the repeated video frames between the current sampled video and the next sampled video, and determine a third hidden vector corresponding to the repeated video frames from the first hidden vector, and determine a fourth hidden vector corresponding to the repeated video frames from the second hidden vector; Based on the third hidden vector and the fourth hidden vector, denoise the current sampled video again; Take the next sampled video as the new current sampled video, and jump to the step of denoising the current sampled video, so as to denoise each of the sampled videos.

12. The video generation method according to claim 11, wherein The denoising of the current sampled video again based on the third hidden vector and the fourth hidden vector includes: Determine a random video frame position from the repeated video frames, and determine a corresponding first video frame in the current sampled video according to the random video frame position; Determine a corresponding second video frame in the next sampled video according to the next video frame position of the random video frame position; Determine a first vector to be spliced corresponding to the first video frame from the third hidden vector, and determine a second vector to be spliced corresponding to the second video frame from the fourth hidden vector; Splice the first vector to be spliced and the second vector to be spliced to obtain a spliced vector, and denoise the current sampled video according to the spliced vector.

13. An electronic device, characterized in that, It includes: A memory for storing a computer program; A processor for implementing the steps of the video generation method according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, wherein the computer program, when executed by a processor, implements the steps of the video generation method according to any one of claims 1 to 12.

15. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the video generation method according to any one of claims 1 to 12.

Images