Video generation method based on large model

Abstract

The invention discloses a video generation method based on a large model, and provides a video generation model comprising a video compression network based on a variational auto-encoder and a user behavior model based on deep learning, and the time and space dimensions of video data are reduced through an encoder of the video compression network. The video is segmented into a space-time potential path; performing feature extraction on the video data through a user behavior model based on deep learning, and generating an embedded vector representing video content; and a decoder of the video compression network is used for recovering a reconstructed video with continuous video frames, so that a high-resolution and high-quality video is generated. According to the invention, large-scale video data can be efficiently processed, and high-quality video content with high resolution and high frame rate can be generated at the same time.

Description

Technical Field

[0001] The present invention relates to the field of video generation, and in particular to a video generation method based on a large model. Background Art

[0002] With the rapid development of artificial intelligence technology, video content generation has become a hot topic of research. Under the current limited technology, generating high-quality and high-resolution content remains a challenge. Existing video generation methods mainly include two categories: traditional computer graphics methods and deep learning-based methods. Traditional computer graphics methods rely on complex algorithms and hand-crafted models. Although they can generate high-quality images and videos, they have a long production cycle, high cost, and are difficult to adapt to diverse video generation needs. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention proposes a video generation method based on a large model, which can generate high-quality, high-resolution, and long-time sequence videos. The specific technical solution is as follows:

[0004] A video generation method based on a large model comprises the following steps:

[0005] Step 1: Preprocess the original video and crop it into a video dataset of uniform size;

[0006] Step 2: construct a video generation model, and use the video data set of uniform size to train the video generation model; the video generation model includes two parts: a video compression network based on a variational autoencoder and a user behavior model based on deep learning; the video compression network based on the variational autoencoder includes an encoder and a decoder, the encoder is used to compress the input original video data in space and time to achieve effective encoding of the video data, and then cut the video that has completed spatial compression and temporal compression into spatiotemporal potential patches consisting of spatial coordinates x, y and time coordinates t for embedding; the user behavior model based on deep learning is used to extract features from the video data and generate an embedding vector representing the video content; the decoder decodes the time dimension and spatial dimension of the embedding vector to restore the reconstructed video with continuous video frames;

[0007] Step 3: Generate a video using the video generation model trained in step 2.

[0008] Furthermore, the user behavior model based on deep learning is a deep learning model based on the Transformer algorithm, including an input layer, a position coding and spatiotemporal coding concatenation layer, a Transformer encoder layer, an embedding vector generation layer, and an output layer;

[0009] The input layer is used to receive the spatiotemporal potential patches output by the encoder;

[0010] The position coding and spatiotemporal coding concatenation layer is used to fuse the position coding, spatiotemporal coding and spatiotemporal latent patch embedding in the spatiotemporal latent patches to form a coding vector containing both temporal information and spatial information;

[0011] The Transformer encoder layer is used to capture the temporal dependency between video frames through the self-attention mechanism of the encoding vector containing both temporal information and spatial information, and extract the deep features of the video content to output a high-dimensional feature vector;

[0012] The embedding vector generation layer is used to reduce the dimension of the high-dimensional feature vector to generate embedding vectors for video generation tasks, and these embedding vectors accurately represent the feature information of the video content;

[0013] The output layer is used to output the embedding vector for the video generation task to the decoder of the video compression network;

[0014] Furthermore, when the video generation model is trained, the loss function includes the loss function L during video compression network training. recon and the loss function L of the user behavior model based on deep learning MSE These two parts.

[0015] Furthermore, the loss function L recon It is used to measure the difference between the reconstructed data output by the decoder and the original input data. The specific calculation formula is as follows:

[0016]

[0017] Where p is the given z i Data point x i The conditional probability of .

[0018] Furthermore, the loss function L MSE It is the mean square error, which is used to measure the average value of the square of the difference between the predicted value and the true value. The specific calculation formula is as follows:

[0019]

[0020] Among them, y i is the true value, is the predicted value and n is the number of samples.

[0021] Furthermore, the video generation model adopts a multi-stage training strategy during training. In the initial stage, low-resolution video data is used for training, and then the resolution of the data is gradually increased until the video generation model can generate high-resolution, high-frame rate videos.

[0022] A video generation device based on a large model includes one or more processors for implementing a video generation method based on a large model.

[0023] An electronic device, comprising:

[0024] one or more processors;

[0025] A storage device is used to store one or more programs, and when the one or more programs are executed by the electronic device, the electronic device implements a video generation method based on a large model.

[0026] A computer-readable storage medium stores a program, which, when executed by a processor, implements a video generation method based on a large model.

[0027] The beneficial effects of the present invention are as follows:

[0028] The present invention solves the challenges of the existing technology in generating high-quality, high-resolution video content by adopting a large-model video generation method based on deep learning VAE and Transformer. The time dimension of the video data is reduced through the video compression network, and the video is divided into time-space potential patches, and the video content is reconstructed after the information in the Transformer modeling video is used. The present invention can efficiently process large-scale video data and generate high-quality video content with high resolution and high frame rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 4 is a flowchart of the operation of the video compression network (VAE) in an embodiment of the present invention.

[0030] Figure 2 It is a schematic diagram of dividing video content into patches in an embodiment of the present invention.

[0031] Figure 3 It is a block architecture diagram of the Transformer in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be described in detail below based on the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more clear. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0033] like Figure 1 and2 As shown, the video generation method based on the large model of this embodiment includes the following steps:

[0034] Step 1: Preprocess the original video and crop it into a video dataset of uniform size.

[0035] Step 2: Build a video generation model and use the video dataset of uniform size obtained in step 1 to train the video generation model.

[0036] The video generation model consists of two parts: a video compression network based on a variational autoencoder and a user behavior model based on deep learning. The video compression network based on the variational autoencoder (VAE) includes an encoder and a decoder, such as Figure 1 As shown in the figure, the main task of the encoder is to convert the input video data into a low-dimensional latent representation. Specifically, it takes the original video data as input and outputs a low-dimensional vector containing key information, which can be regarded as a "compressed" version of the video data. The decoder is responsible for receiving this low-dimensional vector and trying to restore it to the original video data. The encoder and decoder are connected through the latent space, which is the core of the VAE model. In this space, the model can learn the latent representation of the video data, and the decoder uses this low-dimensional vector to reconstruct the video frame, thereby realizing the generation and reconstruction of video data. These two parts work together to form the overall architecture of the VAE network. In the VAE network, spatial compression and temporal compression are usually performed simultaneously. When processing video frames, the encoder will consider both the spatial information within the frame and the temporal information between frames.

[0037] In this embodiment, the encoder is used to perform spatial and temporal compression on the input raw video data to achieve effective encoding of the video data, and then segment the spatially and temporally compressed video into spatiotemporal latent patches consisting of spatial coordinates x, y and temporal coordinates t for embedding. The role of these patches in the model is similar to that of transformer tokens. This patch-based representation can handle videos and images of different resolutions, durations, and aspect ratios. Subsequently, the patches need to undergo a tokenization process, that is, they are converted into one-dimensional embeddings using a DNN, and then fed into a diffusion model under the transformer architecture.

[0038] The user behavior model based on deep learning is used to extract features from video data and generate embedding vectors representing video content. The user behavior model based on deep learning is a deep learning model based on the Transformer algorithm, including an input layer, a position encoding and spatiotemporal encoding concatenation layer, a Transformer encoder layer, an embedding vector generation layer, and an output layer. The Transformer Block architecture is shown in the figure below. Figure 3 shown.

[0039] Wherein, the input layer is used to receive the spatiotemporal potential patches output by the encoder;

[0040] The positional coding and spatiotemporal coding concatenation layer is used to fuse the positional coding, spatiotemporal coding, and spatiotemporal latent patch embeddings in the spatiotemporal latent patches to form a coding vector that contains both temporal and spatial information. In the concatenation process of the positional coding and spatiotemporal coding, the two are simply concatenated together to form a new vector. This new vector contains both the temporal position information of the video frame and its spatial position information, which helps the model better understand the dynamic changes and spatial structure of the video.

[0041] The Transformer encoder layer is used to capture the temporal dependency between video frames through the self-attention mechanism of the encoding vector containing both temporal and spatial information, extract the deep features of the video content, and output a high-dimensional feature vector;

[0042] The embedding vector generation layer is used to reduce the dimensionality of high-dimensional feature vectors and generate embedding vectors for video generation tasks. These embedding vectors accurately represent the feature information of the video content.

[0043] The output layer is used to output the embedding vector for the video generation task to the decoder of the video compression network;

[0044] The decoder decodes the temporal and spatial dimensions of the embedded vector according to the embedded vector of the video generation task, and recovers the reconstructed video with continuous video frames.

[0045] The data transmission process of the entire deep learning-based user behavior model is clear and efficient. From the input layer to the concatenation layer, to the Transformer encoder layer and the embedding vector generation layer, each layer receives the output of the previous layer as input and generates its own output for the next layer to use.

[0046] The loss function of the video generation model during training includes the loss function L of the video compression network recon and the loss function L of the user behavior model based on deep learning MSE These two parts, the loss function L reconIt is used to measure the difference between the reconstructed data output by the decoder and the original input data. The specific calculation formula is as follows:

[0047]

[0048] Where p is the given z i Data point x i The conditional probability of .

[0049] Loss function L MSE It is the mean square error, which is used to measure the average value of the square of the difference between the predicted value and the true value. The specific calculation formula is as follows:

[0050]

[0051] Among them, y i is the true value, is the predicted value and n is the number of samples.

[0052] The training goal of the video compression network based on the variational autoencoder is to maximize the log-likelihood of the input data while ensuring that the distribution of the latent space is close to the prior distribution (usually a normal distribution). This is achieved by minimizing the reconstruction error and regularizing the distribution of the latent space. Such structural characteristics enable VAE to learn a continuous and smooth potential representation of the input data, which is very useful for generating new data samples.

[0053] During the training process of the deep learning-based user behavior model, position encoding is used to provide the Transformer with temporal or spatial position information, so that the model can perceive the order and context of the video frames. Subsequently, through forward propagation, the model learns to transform the input frames into high-dimensional embedding vectors, which encode the feature information of the video frames. The backward gradient backpropagation process then adjusts the parameters of the user behavior model to minimize the reconstruction error and ensure that the generated embedding vector can accurately represent the video content. This process involves optimization algorithms such as stochastic gradient descent (SGD) or its variants, as well as possible regularization techniques to prevent overfitting and improve the generalization ability of the model.

[0054] In addition, when training the video generation model, the resolution and frame rate of the model-generated videos are gradually improved through a multi-stage training strategy. In the initial stage, lower-resolution data is used for training, and then the resolution of the data is gradually increased until the model can generate high-resolution, high-frame-rate videos.

[0055] Step 3: Generate a video using the video generation model trained in step 2.

[0056] Corresponding to the above-mentioned embodiment of the video generation method based on the large model, the present invention also provides an embodiment of a video generation device based on the large model. The video generation device based on the large model includes one or more processors for implementing the video generation method based on the large model in the above-mentioned embodiment.

[0057] The embodiment of the video generation device based on a large model of the present invention can be applied to any device with data processing capabilities, and the arbitrary device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located to read the corresponding computer program instructions in the non-volatile memory into the memory and run them. From the hardware level, in addition to the processor, memory, network interface, and non-volatile memory, the video generation device based on a large model of the present invention, in the embodiment, any device with data processing capabilities where the device is located, can also include other hardware, which is usually based on the actual function of the arbitrary device with data processing capabilities, and will not be described in detail.

[0058] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0059] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0060] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the large model-based video generation method in the above embodiment is implemented.

[0061] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0062] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. A video generation method based on a large model, characterized in that: The following steps are involved: Step 1: Preprocess the original video and crop it into a video dataset of uniform size; Step 2: construct a video generation model, and use the video data set of uniform size to train the video generation model; the video generation model includes two parts: a video compression network based on a variational autoencoder and a user behavior model based on deep learning; the video compression network based on the variational autoencoder includes an encoder and a decoder, the encoder is used to compress the input original video data in space and time to achieve effective encoding of the video data, and then cut the video that has completed spatial compression and temporal compression into spatiotemporal potential patches consisting of spatial coordinates x, y and time coordinates t for embedding; the user behavior model based on deep learning is used to extract features from the video data and generate an embedding vector representing the video content; the decoder decodes the time dimension and spatial dimension of the embedding vector to restore the reconstructed video with continuous video frames; Step 3: Generate a video using the video generation model trained in step 2.

2. The video generation method based on a large model according to claim 1, characterized in that: The user behavior model based on deep learning is a deep learning model based on the Transformer algorithm, including an input layer, a position coding and spatiotemporal coding concatenation layer, a Transformer encoder layer, an embedding vector generation layer, and an output layer; The input layer is used to receive the spatiotemporal potential patches output by the encoder; The position coding and spatiotemporal coding concatenation layer is used to fuse the position coding, spatiotemporal coding and spatiotemporal latent patch embedding in the spatiotemporal latent patches to form a coding vector containing both temporal information and spatial information; The Transformer encoder layer is used to capture the temporal dependency between video frames through the self-attention mechanism of the encoding vector containing both temporal information and spatial information, and extract the deep features of the video content to output a high-dimensional feature vector; The embedding vector generation layer is used to reduce the dimension of the high-dimensional feature vector to generate embedding vectors for video generation tasks, and these embedding vectors accurately represent the feature information of the video content; The output layer is used to output the embedding vector for the video generation task to the decoder of the video compression network.

3. The video generation method based on a large model according to claim 1, characterized in that: When the video generation model is trained, the loss function includes the loss function L during video compression network training. recon and the loss function L of the user behavior model based on deep learning MSE These two parts.

4. The video generation method based on a large model according to claim 1, characterized in that: Loss function L recon It is used to measure the difference between the reconstructed data output by the decoder and the original input data. The specific calculation formula is as follows: Where p is the given z i Data point x i The conditional probability of .

5. The video generation method based on a large model according to claim 3 is characterized in that: Loss function L MSE It is the mean square error, which is used to measure the average value of the square of the difference between the predicted value and the true value. The specific calculation formula is as follows: Among them, y i is the true value, is the predicted value and n is the number of samples.

6. The method for generating a video based on a large model according to claim 3, characterized in that: The video generation model adopts a multi-stage training strategy during training. In the initial stage, low-resolution video data is used for training, and then the resolution of the data is gradually increased until the video generation model can generate high-resolution, high-frame rate videos.

7. A video generation device based on a large model, characterized in that: It comprises one or more processors, used to implement the large model-based video generation method described in any one of claims 1 to 6.

8. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the electronic device, enables the electronic device to implement the large model-based video generation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the large model-based video generation method described in any one of claims 1 to 6 is implemented.

Images