Interactive video generation method

Abstract

The invention provides an interactive video generation method, and relates to the technical field of deep learning, and the method comprises the steps: obtaining training data which comprises video data and interaction information corresponding to the video data; the video data consists of a plurality of frames of low-resolution images; obtaining image hidden layer features of the initial frame from the training data; performing feature conversion processing according to the image hidden layer feature and the interaction information of the corresponding frame to obtain a first frame image and a target image hidden layer feature; performing image reconstruction processing according to the hidden layer features of the target image to obtain a reconstructed low-resolution image; performing image super-resolution processing according to the reconstructed low-resolution image to obtain pre-generated image frames until the pre-generated image frames of each frame number in the corresponding training data are obtained, and obtaining a video generation model; and processing the input data by adopting the video generation model to obtain a target video. According to the method, the problem that real-time interaction adjustment cannot be carried out in the existing video generation technology is solved.

Description

[0001] This application is a divisional application. The application number of the original application is 202410939073.0, and the original application date is July 15, 2024. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present invention relates to the field of deep learning technology, and in particular to an interactive video generation method. Background Art

[0003] With the rapid development of information technology, video generation and processing technology has become a major research hotspot in the field of contemporary science and technology. Among the many video processing technologies, interactive video generation methods are particularly eye-catching. This method can not only realize the basic functions of traditional video production, but also enable users to interact with video content in real time, greatly enriching the scenarios and possibilities of video applications.

[0004] In the current research, the video generation technology mainly includes the following two options. The first option is to import text information into the neural network, generate a frame of video based on the text information through the neural network, and obtain each subsequent frame in the same way to obtain the video image corresponding to the text information; the second option is to obtain the time series noise by importing the time series noise into the 3D generation model to obtain the video image.

[0005] However, whether it is solution 1 or solution 2, the generated video content is fixed, that is, each frame of the image is predetermined and cannot be dynamically adjusted according to real-time interactive operations. Secondly, most existing technologies output in the form of video files and cannot generate real-time video streams, which is particularly insufficient in scenarios that require instant feedback and high interactivity. Summary of the invention

[0006] The present application provides an interactive video generation method and system to solve the problem that existing video generation technology cannot perform real-time interactive adjustment.

[0007] The method comprises:

[0008] Acquire training data, wherein the training data includes video data and interaction information corresponding to the video data; the video data is composed of a plurality of frames of low-resolution images;

[0009] Acquire the image hidden layer features of the initial frame from the training data;

[0010] Performing feature conversion processing according to the image hidden layer features and the interactive information of the corresponding frame to obtain the first frame image and the target image hidden layer features;

[0011] Performing image reconstruction processing according to the hidden layer features of the target image to obtain a reconstructed low-resolution image;

[0012] Performing image super-resolution processing according to the reconstructed low-resolution image to obtain pre-generated image frames, until the pre-generated image frames corresponding to each number of frames in the training data are obtained, thereby obtaining a video generation model;

[0013] The video generation model is used to process the input data and the target video is obtained.

[0014] Preferably, the step of acquiring the image hidden layer features of the initial frame from the training data comprises:

[0015] Using the training data to train a low-resolution image reconstruction model, wherein the low-resolution image reconstruction model is built into the interactive video generation model;

[0016] Acquire hidden layer features of each frame of image during the process of training the low-resolution image reconstruction model;

[0017] Select any one feature from all the hidden layer features as the image hidden layer feature.

[0018] Preferably, the step of acquiring the image hidden layer features of the initial frame from the training data further comprises:

[0019] After completing the training of the low-resolution image reconstruction model, obtaining the hidden layer features of each frame of the image;

[0020] Sampling is performed from the prior distribution of all the hidden layer features to obtain the image hidden layer features.

[0021] Preferably, the step of performing feature conversion processing according to the interactive information of the hidden layer features of the image and the corresponding frame number includes:

[0022] Acquire the interaction information of the corresponding number of frames according to the hidden layer features of the image;

[0023] The image hidden layer features and the interactive information of the corresponding frame number are imported into the converter of the interactive video generation model for feature conversion processing to obtain the first frame image and the target image hidden layer features.

[0024] Preferably, the step of performing image reconstruction processing according to the hidden layer features of the target image includes:

[0025] The target image hidden layer features are imported into the trained low-resolution image reconstruction model, and image reconstruction processing is performed to obtain the reconstructed low-resolution image.

[0026] Preferably, the interactive video generation model also has a built-in trained super-resolution model, and the step of performing image super-resolution processing based on the reconstructed low-resolution image includes:

[0027] The reconstructed low-resolution image is imported into the super-resolution model for super-resolution processing to obtain the pre-generated image frame.

[0028] Preferably, the converter comprises an interactive information encoder, a conversion encoder and a conversion decoder, and the method further comprises:

[0029] Importing the interactive information corresponding to the low-resolution image into the interactive information encoder for interactive encoding to obtain interactive hidden features, and transmitting the interactive hidden features to the transform decoder through the transform encoder;

[0030] The image hidden layer features are introduced into the conversion decoder, and the target hidden layer features of the corresponding image frame are generated by the conversion decoder according to the interactive hidden features and the image hidden layer features.

[0031] Preferably, the low-resolution image reconstruction model includes an image reconstruction encoder, and the method further includes:

[0032] The low-resolution image with a corresponding number of frames is imported into the image reconstruction encoder for image reconstruction encoding to obtain the image hidden layer features.

[0033] Preferably, the low-resolution image reconstruction model further includes an image reconstruction decoder, and the method further includes:

[0034] The target hidden layer features are introduced into the image reconstruction decoder for image reconstruction decoding to obtain the reconstructed low-resolution image.

[0035] Preferably, the resolution of the pre-generated image frame is the same as the resolution of the video data.

[0036] As can be seen from the above content, the present application provides an interactive video generation method, the method comprising obtaining training data, the training data comprising video data and interactive information corresponding to the video data; the video data is composed of several frames of low-resolution images; obtaining the image hidden layer features of the initial frame from the training data; performing feature conversion processing according to the image hidden layer features and the interactive information of the corresponding frame to obtain the first frame image and the target image hidden layer features; performing image reconstruction processing according to the target image hidden layer features to obtain a reconstructed low-resolution image; performing image super-resolution processing according to the reconstructed low-resolution image to obtain a pre-generated image frame, until the pre-generated image frame corresponding to each frame number in the training data is obtained, and a video generation model is obtained; the video generation model is used to process the input data to obtain the target video. The present application solves the problem that the existing video generation technology cannot perform real-time interactive adjustment through the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 A flowchart of a real-time interactive video generation method for this application;

[0039] Figure 2 A schematic diagram of data processing for training a low-resolution image reconstruction model and an image super-resolution model in a real-time interactive video generation method of the present application;

[0040] Figure 3 A schematic diagram of data processing for training an interactive video generation model in a real-time interactive video generation method of the present application;

[0041] Figure 4 A schematic diagram of data processing for training a converter in an interactive video generation model for this application;

[0042] Figure 5 A schematic diagram of a real-time interactive video generation system according to the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] In a specific interactive scenario, the acquired continuous interactive operations are used to generate continuous video images, and the content contained in the generated video images matches the interactive operations. The above interactive scenarios include but are not limited to using keyboard or mouse clicks, game controller operations, voice control, motion capture of somatosensory devices, motion capture using photographic equipment, or brain-computer interfaces. The above video images and content are generated by the above interactions, and the content changes with the changes in the above interactions, including but not limited to game images, scene guides and other video content.

[0045] For example, in a certain game, the keyboard direction keys are used to provide operations, and the game screen generates corresponding images as the keyboard input changes; continuous keyboard input and corresponding image content are collected to train the interactive video generation model; continuous keyboard interactions are input into the interactive video generation model to generate continuous image content, that is, video content; this video content is related to a specific scene, that is, a certain interactive video generation model can generate video content for a specific scene; if video content for a new scene needs to be generated, it is necessary to re-collect data and train a new interactive video generation model.

[0046] The existing technologies for generating videos currently include existing embodiment 1 with publication number CN114598926A, paper 1 entitled “Generating Long Videos of Dynamic Scenes”, and paper 2 entitled “Video Diffusion Models”.

[0047] According to Example 1, the existing video generation process includes (1) inputting text information into a Transformer neural network Decoder model, encoding the text information, and generating encoding features of a corresponding frame image; (2) using a VQVAE neural network Decoder model to decode the encoding features of the frame image into a current frame image; (3) downsampling the generated current frame image, inputting the VQVAE neural network Encoder model, and obtaining encoding information of the generated image; (4) inputting the text information and the encoding information of the generated image into a Transformer neural network Decoder model, and obtaining encoding features of the corresponding next frame image; (5) repeating steps (2) to (4) until a preset number of frames is reached, and a complete video is output.

[0048] According to Paper 1, Paper 1 is an unconditional video generation method. The process of generating videos in Paper 1 is as follows: (1) Randomly sample temporal noise, and input the temporal noise into a 3D generator model similar to StyleGAN2 and StyleGAN3 through a mapping network. The 3D generator model upsamples the initial input from the temporal dimension and the spatial dimension to generate a low-resolution video; (2) Input the low-resolution video generated in step (1) into a video super-resolution model to obtain a high-resolution generated video.

[0049] According to the second paper, the second paper uses text information as a condition for video generation, and can also perform unconditional video generation. The method of generating videos in the second paper uses a diffusion model with 3D-UNet as the backbone, samples noise from a standard Gaussian distribution, and gradually denoises it to finally obtain a generated video. In the process of gradual denoising, a language model, such as pre-trained BERT, CLIP, etc., can be used to extract text features from given text information, and the text features are injected into 3D-UNet using a self-attention mechanism to provide text conditional guidance for video generation. Of course, it is also possible to do unconditional video generation without using text information.

[0050] According to the above scheme, it can be known that: Example 1 is conditional video generation using text information, Paper 1 is unconditional video generation without using other information, and Paper 2 is conditional video generation using text information or unconditional video generation without using other information. In the above method, given text information or a given initial state, the generated video content is fixed, that is, the image of each frame of the generated video is fixed. Each frame of the generated video is only related to the initial input. If the input changes during the generation process, that is, interaction occurs, the above method cannot handle it. At the same time, the output of Papers 1 and 2 is in the form of video, and it is impossible to generate real-time video streams.

[0051] Based on the above content, this embodiment provides the following solution to solve the above problem.

[0052] Figure 1 This is a flowchart of a real-time interactive video generation method for the present application.

[0053] See also Figure 1 It can be seen that this embodiment provides a real-time interactive video generation method, the generation method comprising:

[0054] S100, obtaining training data, the training data including interaction information and video data corresponding to the interaction information. Specifically, in the present embodiment, the training data is continuous interaction information and video data corresponding to the interaction. Here, the interaction information includes but is not limited to keyboard or mouse clicks, game controller operations, voice control, motion capture of somatosensory devices, motion capture using photographic equipment, or brain-computer interface, etc. These interaction information are directly collected using some external devices; or some external devices are used to collect data first, and then the relevant software or algorithm extracts it from the collected data. The video data corresponding to the interaction refers to the video generated by the above-mentioned interaction process, and the changes in the above-mentioned interaction process can correspond to the changes in the video content. Taking racing games as an example, keyboard operations can control the movement of objects in the game screen, and when the keyboard operation changes, the content of the game screen will change accordingly.

[0055] When acquiring training data, interactive operation information and video image data must be recorded simultaneously. The longer the video used for training, the better, not less than 10 minutes, with no specific restrictions. The video resolution can be a common resolution, such as 360P, 480P, 720P, 1080P, etc., but it is not mandatory. The video frame rate can be the currently commonly used video frame rate, such as 25 frames / second or 30 frames / second, and it is not mandatory. At the same time, record the interactive operation information synchronized with the video, the frame rate of the interactive operation information is not lower than the video frame rate, and the frame rate of the interactive operation information is an integer multiple of the video frame rate. For example, if the recorded video frame rate is 30 frames / second, the frame rate of the recorded interactive operation information can be 30 frames / second, 60 frames / second, 90 frames / second, 120 frames / second, etc.

[0056] S200, preprocessing the training data to obtain target training data. Specifically, in this embodiment, in order to facilitate subsequent model training, the training data needs to be preprocessed first, wherein the step of preprocessing the training data includes:

[0057] The video data is downsampled to obtain a low-resolution image, wherein the ratio of reducing the resolution by the downsampling is determined according to the size of the frame image of the video data; the low-resolution image is cropped to obtain a plurality of groups of video clips, each of the video clips includes T frames, two adjacent video clips are separated by 2 / T frames, and two adjacent video clips overlap by 2 / T frames; the corresponding interaction information is recorded for each of the video clips to obtain the target training data, wherein the frame rate of the interaction information recorded in each of the video clips is not less than the number of frames corresponding to the video clip, and the frame rate of the interaction information recorded in each of the video clips is an integer multiple of the number of frames corresponding to the video clip.

[0058] For example, to train the image reconstruction model and the image super-resolution model, it is necessary to extract all frames of images from the captured video. The resolution of these frame images is the same as the resolution of the video, which is recorded as the original resolution image x. The original resolution image x is downsampled by a certain ratio to obtain a low-resolution image x. L The downsampling ratio needs to be determined according to the size of the frame image. For example, if the video frame image is 360P or 480P, the downsampling ratio is 2, that is, the low-resolution image x after downsampling L The length and width are the original resolution image x If the video frame image is 720P or 1080P, the downsampling ratio is 4, that is, the low-resolution image after downsampling is x L The length and width are the original resolution image x The higher the resolution of the original image, the higher its downsampling ratio.

[0059] In order to train an interactive video generation model, the original video needs to be cut into several segments. The image part of each video segment is recorded as v, and v consists of a set of frame images [x 1 ,...,x t ,...,x T ], T is the number of frames contained in each video segment, and t is the index of the frame image in the video segment. When the video is cut into segments, each segment contains T frames, and the interval between adjacent segments is frames, then the adjacent fragments have The interactive operation information corresponding to the video segment v is I = [I 1 ,...,I t ,...,I T ]. Each frame image x t There are corresponding interactive operations I t Since the frame rate of the interactive operation information is not lower than the video frame rate during data recording, and the frame rate of the interactive operation information is an integer multiple of the video frame rate (denoted as N), each frame of the video image corresponds to a set of interactive operation Where n is x t The index of the interactive operation information corresponding to the frame image, is an interactive information vector. Due to the diversity of interactive forms, It may be discrete or continuous. For example, when the interaction information is collected from keyboard or mouse clicks or game controller operations, the interaction information is expressed as a discrete value of whether the button is clicked or not. is a set of one-hot encoded vectors; when the interactive information is collected by voice control, motion capture of somatosensory devices, motion capture using photographic equipment, or brain-computer interface, the interactive information is represented by a set of continuous values. The characteristic number of a vector, then I t is an f×N matrix. In order to unify the discrete interactive operation information and the continuous interactive operation information, in this embodiment, I t Gaussian smoothing is performed on the time dimension (i.e., N) to convert discrete interaction information into continuous interaction information, so that continuous interaction information can be obtained for training and inference under different interaction modes.

[0060] For some more complex interaction methods, such as voice control, motion capture using somatosensory devices, motion capture using photographic equipment, or brain-computer interfaces, the raw data collected by them may include sound signals, images, electrical signals, etc. The process of processing the raw data collected by each complex interaction method is not included in the data preprocessing process of this embodiment. The data preprocessing of the interactive operation information in this embodiment uses the information generated by these interaction methods themselves after processing their raw data. For example, when using voice control as an interaction method, this embodiment does not directly use sound signals, but uses some existing methods to process sound signals, and uses the processed sound features as interactive operation information.

[0061] S300, model training is performed according to the target training data, and the model training includes a model pre-training process and a video pre-generation process; the model training process includes the training of a low-resolution image reconstruction model and the training of an image super-resolution model; the video pre-generation process includes video pre-generation according to an interactive video generation model, so as to complete the training of the interactive video generation model, wherein the interactive video generation model has the trained low-resolution image reconstruction model and the image super-resolution model. Specifically, in this embodiment, step S300 is a model training step, and the model training process is divided into two stages: (1) a pre-training process, which includes the training of a low-resolution image reconstruction model and the training of an image super-resolution model; (2) training of an interactive video generation model, generating images frame by frame, that is, merging all generated images to generate a video, or outputting the image to a video stream in real time. There is no order requirement for the training of the two models in the pre-training process. The pre-training process must be performed before the training of the interactive video generation model.

[0062] Figure 2 A schematic diagram of data processing for training a low-resolution image reconstruction model and an image super-resolution model in a real-time interactive video generation method of the present application.

[0063] See also Figure 2 It can be seen that training a low-resolution image reconstruction model includes the following schemes.

[0064] The low-resolution image reconstruction model consists of an image encoder E Mand image decoder D M It is composed of convolutional layers, fully connected layers, activation layers, pooling layers, normalization layers and other basic neural network structures. Some existing network structures can be used, such as AutoEncoder, VAE, VQVAE and other neural networks with encoder and decoder structures, or the basic structure can be used to design the image reconstruction model. There is no requirement for the specific structure of the neural network here.

[0065] The low-resolution image reconstruction model uses the low-resolution image x L As input, after the image encoder E M , get the input low-resolution image x L The hidden code z x , the hidden layer code z x Input image decoder D M , obtain the reconstructed low-resolution image Hope to reconstruct the image With the input image x L In this process, the image encoder E M The resulting hidden code z x It can be used to represent input images and subsequently used for training and inference of interactive video generation models.

[0066] To train the low-resolution image reconstruction model, use the real low-resolution image x L As input, it also serves as supervision data. After the image encoder E M and image decoder D M After that, the model outputs the reconstructed low-resolution image Calculate the reconstruction loss function The reconstruction loss function is based on the MAE loss (L 1 loss), MSE loss (L 2 loss), perceptual loss, image similarity loss, or any other loss function that can characterize image differences. After calculating the reconstruction loss function, the back propagation process is performed. According to the calculated loss function, the image encoder E M and image decoder D M The gradient of the parameters of each component in the low-resolution image reconstruction model is calculated, and the gradient descent method is used to optimize and update the parameters of each component in the low-resolution image reconstruction model. The model training process includes several rounds, and each round executes the above model training process until the training termination condition is met and the model training is completed.

[0067] It should be noted that when using different neural networks, the loss functions that need to be calculated may be different.

[0068] See also Figure 2 It can be seen that training the image super-resolution model includes the following schemes.

[0069] The image super-resolution model S can process the input low-resolution image to obtain a high-resolution image corresponding to the content of the low-resolution image to improve the clarity of the input low-resolution image. The image super-resolution model S is composed of basic neural network structures such as convolutional layers, fully connected layers, activation layers, pooling layers, and normalization layers. You can use an existing neural network for super-resolution tasks as a super-resolution generation model, or you can use the basic structure to design a super-resolution generation model yourself. There is no requirement for the specific structure of the neural network here. Existing image super-resolution models include but are not limited to HAT, SwinIR, LTE, etc.

[0070] The image super-resolution model S contains a series of upsampling layers, using the low-resolution image x L As input, after upsampling, the reconstructed original resolution image is output Hope to reconstruct the image is as identical as possible to the true original resolution image x.

[0071] To train the image super-resolution model S, we use the real low-resolution image x L As input, the corresponding original resolution image x is used as supervision data. After the image super-resolution model S, the reconstructed original resolution image is output Calculate the reconstruction loss The reconstruction loss function can use MAE loss (L 1 loss), MSE loss (L 2 Loss), perceptual loss, image similarity loss or any other loss function that can characterize image differences. Optimize the reconstruction loss function to ensure that the reconstructed image output by the image super-resolution model S has the same content as the true original resolution image.

[0072] By optimizing the image discriminator D S , judge whether the input image is a real image or a reconstructed image, so that the clarity of the reconstructed image output by the image super-resolution model S is close to the real image. The image discriminator can optimize the super-resolution generation model S, so that the image output by the super-resolution model S is closer to the real image. The image discriminator is composed of basic neural network structures such as convolutional layer, fully connected layer, activation layer, pooling layer, and normalization layer. You can use the existing discriminator structure of the generative adversarial network, or you can use the basic structure to design the discriminator yourself.

[0073] Image Classifier D S It is used to determine whether the input image is a real original resolution image or a reconstructed original resolution image. It is a model for binary classification. Therefore, the commonly used cross entropy loss function can be used. Cross entropy loss function L SAs shown in formula [1], where x i is the image input to the image discriminator, y i is the label of this image. When the image x is input to the image discriminator i When it is a true original resolution image, y i is 1; when the image input to the image discriminator is the reconstructed original resolution image, y i is 0. s (x i ) is the output after the image is input into the image discriminator, and its value represents the probability that the input image is a real image. In addition to using the cross entropy loss function for discriminator classification, other loss functions for classification can also be used.

[0074] L S =-y i log(D s (x i ))-(1-y i )log(1-D s (x i )) (1)

[0075] Calculate the above reconstruction loss L R and the cross entropy loss L of the discriminator S After that, the back propagation process is performed. According to the calculated loss function, the gradient of the parameters of each component in the model is calculated, and the gradient descent method is used to optimize and update the parameters of each component in the model. It should be noted that the above reconstruction loss L R Only used to optimize the image super-resolution model S parameters, discriminator loss L S It is also used to optimize the image super-resolution model S parameters and image discriminator D S Parameters. The model training process includes several rounds, and each round executes the above model training process until the training termination condition is met and the model training is completed.

[0076] Figure 3 A schematic diagram of data processing for training an interactive video generation model in a real-time interactive video generation method of the present application.

[0077] See also Figure 3 It can be seen that the scheme for training an interactive video generation model is as follows.

[0078] Among them, the video pre-generation process (i.e., training an interactive video generation model) includes generating image hidden layer features corresponding to the initial frame of the low-resolution image; inputting the image hidden layer features and the interactive information of the corresponding low-resolution image into the converter of the interactive video generation model to generate target hidden layer features corresponding to the first frame image; reconstructing the target hidden layer features to obtain a reconstructed low-resolution image; performing super-resolution processing on the reconstructed low-resolution image to obtain a pre-generated first frame image corresponding to the pre-generated video, and thereby generating each frame image corresponding to the pre-generated video, and the resolution of the pre-generated first frame image is the same as the resolution of the video data.

[0079] in, Figure 3 Transformer encoder-decoder M V and interactive encoder E I See as a converter in the above content.

[0080] The interactive video generation model includes components such as the interactive information encoder, the Transformer encoder-decoder, the image encoder, the image decoder, and the image super-resolution model. The image encoder, the image decoder, and the image super-resolution model have been obtained through the above pre-training process. Therefore, the process of training the interactive video generation model only requires training the interactive information encoder and the Transformer encoder-decoder.

[0081] Figure 4 Schematic diagram of the data processing for the converter in the interactive video generation model trained for this application.

[0082] See also Figure 4 It can be seen that, further, in some embodiments, the converter includes an interactive information encoder, a conversion encoder and a conversion decoder. Therefore, the step of generating the target hidden layer features corresponding to the first frame image also includes inputting the interactive information corresponding to the low-resolution image into the interactive information encoder to obtain interactive hidden features corresponding to the interactive information of the low-resolution image; inputting the image hidden layer features into the conversion decoder, transmitting the interactive hidden features to the conversion decoder through the conversion encoder, and generating the target hidden layer features of the corresponding image frame through the conversion decoder according to the interactive hidden features and the image hidden layer features.

[0083] In order to avoid confusion, the text description will be Figure 4 Transformer encoder ME in v It is called the conversion encoder, and the Transformer decoder MD v It is called a transform decoder.

[0084] Exemplarily, the mutual information encoder E I Using pre-processed interactive information I t As input, network feedforward and nonlinear transformation are performed to obtain the hidden layer features of interactive operation information. I It is composed of basic neural network structures such as convolutional layer, fully connected layer, activation layer, pooling layer, and normalization layer. Figure 3 and Figure 4 In the interactive operation information encoder E I The input is interactive operation information I t , that is, the interactive operation information corresponding to the tth frame of the video to be generated. In fact, according to different video generation tasks, the interactive information encoder E I The input can also be in various forms.

[0085] The above different video tasks are divided into: (1) Offline tasks. Given a set of known interactive operation information I = [I 1 ,···,I t ,···,I T ], where T is the number of frames to be generated, and t is the tth frame of the generated video. Using the known interactive operation information I, directly generate the corresponding video v. (2) Real-time task. Given the interactive information of the current frame of the historical frame I = [I 1 ,···,I t ], generate the video frame corresponding to the current time t The generated video frames are output in the form of a video stream to achieve real-time video generation.

[0086] For offline tasks, since the interactive operation information corresponding to all frames is known, the interactive information encoder E I We can use only the interactive operation information corresponding to the tth frame, or we can use the interactive information corresponding to the tth frame and several frames before and after it, such as [···,I t-2 ,I t-1 ,I t ,I t+1 ,I t+2 ,···]. When using multi-frame interactive operation information, it is only necessary to splice the interactive operation information of multiple frames in the time dimension.

[0087] For real-time tasks, since only the interactive operation information corresponding to the historical frame and the current frame is known, the interactive information encoder E I We can use only the interactive operation information corresponding to the tth frame, or we can use the interactive operation information corresponding to the tth frame and several frames before it, such as [···,I t-2 ,I t-1 , It]. When using multi-frame interactive operation information, it is only necessary to splice the multi-frame interactive operation information in the time dimension.

[0088] Transformer encoder-decoder M V The structure is the commonly used Transformer model, which will not be explained in detail here. Transformer encoder ME v Encode the hidden features of the input interactive operation information, and use its internal self-attention mechanism to calculate the importance of each step of the sequential interactive operation information to generate the corresponding frame. Transformer decoder MD v Using Encoder ME v The output of is used as input, and the hidden features of the image generated by the previous frame are used As input. The generated previous frame image is input into the image encoder E M You can get Transformer Decoder MD v The role of is to combine the interactive operation information with the information of the image generated in the previous frame to generate the hidden layer features of the current frame For image decoder D M Generate current frame image The image clarity is then improved through the image super-resolution model S, and finally the image of the current frame is obtained

[0089] Furthermore, after completing the generation of a video, the present embodiment further optimizes the scheme, and the specific scheme is as follows: merging all generated image frames into the pre-generated video; calculating the reconstruction loss function of the pre-generated video and the real video corresponding to the video data, the reconstruction loss function including MAE loss, MSE loss, perceptual loss and image similarity loss; constructing a cross-entropy loss function, the cross-entropy loss function is used to adjust the parameters of the video discriminator, the video discriminator is a component in the interactive video generation model used to adjust the clarity of the pre-generated video to gradually approach the real video; adjusting the parameters of each component in the interactive video generation model through the reconstruction loss function and the cross-entropy loss function; the step of adjusting the parameters of each component in the interactive video generation model through the reconstruction loss function and the cross-entropy loss function includes calculating the gradient of the parameters of each component in the interactive video generation model through the reconstruction loss function, and adjusting the parameters of each component in the interactive video generation model by gradient descent.

[0090] Specifically, in this embodiment, the interactive information encoder E for training the interactive video generation model I, Transformer encoder-decoder M V , using the preprocessed interactive operation information as input and the real video clip v corresponding to the interactive operation as supervision information. After the interactive video generation model, the generated video with the same resolution as the real video clip is output Calculate the reconstruction loss of the generated video and the real video The reconstruction loss function is based on the MAE loss (L 1 loss), MSE loss (L 2 Loss), perceptual loss, image similarity loss, or any other loss function that can characterize image differences. Optimize the reconstruction loss function to ensure that the video generated by the interactive video generation model has the same content as the real video.

[0091] Using video discriminator D V Determine whether the input video is a real video or a generated video, so that the clarity of the video output by the interactive video generation model is close to the real video. Its principle and loss function calculation formula are similar to those of the image discriminator D S Same, no more repetition. Video Identifier D V and image discriminator D S The difference is that the image discriminator D S The input is a 2D image, the video discriminator D V The input is 3D video. Video Identifier D V It is used to determine whether the input video is a real video or a generated video. It is a model for binary classification, so the cross entropy loss function L in the same form as formula [1] can also be used. V To optimize the video discriminator D V .

[0092] Calculate the above reconstruction loss L R and the cross entropy loss L of the discriminator V After that, the back propagation process is performed. According to the calculated loss function, the gradient of the parameters of each component in the interactive video generation model is calculated, and the gradient descent method is used to optimize and update the parameters of each component in the model. It should be noted that the above reconstruction loss L R Only used to optimize the mutual information encoder E I and Transformer encoder-decoder M V , the discriminator loss function L V It is also used to optimize the mutual information encoder E I , Transformer encoder-decoder M V and video discriminator D V Parameters. Image encoder E M , Image decoder D MThe parameters of the image super-resolution model S are not updated during this training process. The model training process includes several rounds, and each round executes the above model training process until the training termination condition is met, completing the training of the interactive video generation model.

[0093] Among them, see Figure 3 It can be known that the specific scheme of performing image reconstruction on the target hidden layer features to obtain the reconstructed low-resolution image is to input the target hidden layer features into an image reconstruction decoder, and perform image reconstruction on the target hidden layer features through the image reconstruction decoder to obtain the reconstructed low-resolution image. The image reconstruction decoder is a decoder of the low-resolution image reconstruction model that has completed training.

[0094] Among them, see Figure 3 It can be known that the specific scheme of performing super-resolution processing on the reconstructed low-resolution image to obtain the pre-generated first frame image corresponding to the pre-generated video is to input the reconstructed low-resolution image into the trained image super-resolution model, and perform super-resolution processing on the reconstructed low-resolution image through the image super-resolution model to obtain each frame image corresponding to the pre-generated video with the same resolution as the video data.

[0095] It should be noted that in order to reflect that the interactive video generation model has the trained low-resolution image reconstruction model, Figure 3 The image encoder E in M and image decoder D M Called image reconstruction encoder and image reconstruction decoder.

[0096] It should be noted that after the first pre-generated image frame is generated, the generation of subsequent image frames depends on the data of the previous image frame. The specific scheme is as follows: when the t-th pre-generated image is generated, the image frame of the t-1 frame is input into the image reconstruction encoder to obtain the image hidden layer features of the t-1 frame image frame; the interaction information corresponding to the t-1 frame image frame is obtained, and the image hidden layer features and interaction information corresponding to the t-1 frame image frame are input into the converter of the interactive video generation model to obtain the target hidden layer features of the t-1 frame image frame; the target hidden layer features of the t-1 frame image frame are input into the image reconstruction decoder, and the image reconstruction decoder is used to reconstruct the target hidden layer features of the t-1 frame image frame to obtain the reconstructed low-resolution image of the t-1 frame image frame; the reconstructed low-resolution image of the t-1 frame image frame is input into the trained image super-resolution model, and the reconstructed low-resolution image of the t-1 frame image frame is super-resolution processed by the image super-resolution model to obtain the t-1 pre-generated image frame.

[0097] Step S300 can be summarized as follows:

[0098] (1) Generate the hidden layer features z of the initial frame 0 There are two generation methods: ① In the process of pre-training the image reconstruction model, the hidden features of all known images are obtained, and the hidden features of an image are randomly selected as z 0 ; ② After pre-training the image reconstruction model, the prior distribution of the hidden layer features is known, and z can be sampled from the prior distribution of the hidden layer features 0 .

[0099] (2) The hidden layer feature z of the initial frame 0 And the interactive operation information I corresponding to the first frame 1 The hidden layer features are input into the Transformer encoder-decoder M V , obtain the hidden layer features used to reconstruct the first frame image Will Input image decoder D M , obtain the reconstructed low-resolution image Will Input the image super-resolution model S to obtain the generated image of original resolution That is, the generation of the first frame of the video is completed.

[0100] (3) To generate the tth frame of the video, first generate the previous frame of the video Input image encoder E M , get the previous frame image The hidden layer code Will And the interactive operation information I corresponding to the current frame t The hidden layer features are input into the Transformer encoder-decoder M V , obtain the hidden layer features used to reconstruct the current frame image Will Input image decoder D M , obtain the reconstructed low-resolution image Will Input the image super-resolution model S to obtain the generated image of original resolution That is, the generation of the tth frame of the video is completed.

[0101] (4) Repeat step (3) until the termination condition is reached. Different video generation tasks have different termination conditions. For offline tasks, the termination condition is generally when the number of generated video frames reaches a preset value. At this time, all generated frames are merged into a video. For real-time tasks, the generated video frames are output to the video stream, and the termination condition is generally that the user terminates the generation process. The termination condition during the training process is that the number of generated video frames reaches the number of frames in the training video clip.

[0102] Exemplarily, this embodiment provides the following model training example:

[0103] (1) Use the keyboard to play a game, and record the keyboard operation while recording the game video. The video length is 30 minutes, the video resolution is 512×512, the video frame rate is 30 frames / second, and the keyboard operation recording frame rate is 300 frames / second, that is, each frame of the video image corresponds to 10 frames of keyboard operation recording.

[0104] (2) Downsample the original video with a resolution of 512×512 to a resolution of 256×256; extract all frame images from the original video with a resolution of 512×512, and extract all frame images from the downsampled video with a resolution of 256×256; crop the 30-minute full video into several video clips, each video clip contains 16 frames of images, and the cropping interval between two adjacent video clips is 8 frames. At the same time, crop the keyboard interaction record data, each video clip contains 160 frames of interaction data, and each frame of image corresponds to 10 frames of interaction data; perform Gaussian smoothing with a mean of 0 and a variance of 3 on the interaction data corresponding to each frame.

[0105] (3) Using the downsampled frame image with a resolution of 256×256 as input data and supervision data, the image reconstruction model VQVAE is trained to obtain the image encoder and image decoder, and the prior distribution of the image hidden layer features is obtained;

[0106] (4) Use the downsampled frame image with a resolution of 256×256 as input data and the original frame image with a resolution of 512×512 as supervision data to train the SwinIR image super-resolution model;

[0107] (5) Use the VQVAE encoder to extract hidden features from the previous frame of generated image (the hidden features of the initial frame are sampled from the prior distribution of the image hidden features obtained by pre-training VQVAE); use the interactive operation information of length 10 corresponding to the current frame to input the interactive information encoder to obtain the interactive information hidden features; input the generated previous frame image hidden features and the interactive information hidden features into the Transformer encoder-decoder to generate the hidden features of the current frame image; input it into the image decoder to generate a low-resolution image of the current frame, that is, its resolution is 256×256; then input the low-resolution image into the pre-trained SwinIR image super-resolution model to obtain an image with an original resolution of 512×512, which is the generated current frame image.

[0108] Repeat process (5) 16 times to obtain the generated video clips. Use the generated video clips and the real video clips to calculate the relevant loss function, which is used to optimize the parameters of the interactive information encoder and the Transformer encoder-decoder until the training is terminated, thus obtaining a trained interactive video generation model.

[0109] S400, generating an interactive video using the trained interactive video generation model. Specifically, in this embodiment, during the process of generating the interactive video, the data processing process of the interactive video generation model is as follows:

[0110] ① The operation process of the interactive information encoder and the Transformer encoder-decoder can be placed in one process, and its input queue uses the output queue of process ④ to establish an output queue for it;

[0111] ②Place the operation process of the image decoder in a process, and its input queue uses the output queue of process ① to establish an output queue for it;

[0112] ③Place the computation process of the image super-resolution model in a process, whose input queue uses the output queue of process ② and establishes an output queue for it;

[0113] ④Place the operation process of the image encoder in a process, and its input queue uses the output queue of process ② to establish an output queue for it;

[0114] ⑤The final video output process is placed in a process, whose input queue uses the output queue of process ③, and each frame of the generated image is output to a video file or video stream.

[0115] The real-time performance of the model inference process can be enhanced by running in parallel.

[0116] Exemplarily, this embodiment provides the following example of generating an interactive video:

[0117] (1) Given a set of interaction operation sequences of known length, whose length is 3000 frames, the corresponding generated video frame number is 300 frames and the video length is 10 seconds.

[0118] (2) The hidden features of the initial frame are sampled from the prior distribution of the hidden features of the image obtained by pre-training VQVAE, and the hidden features of the initial frame and the interactive operation information corresponding to the generation of the first frame image are input into the Transformer encoder-decoder parameters to obtain the hidden features of the first frame image, which are then input into the VQVAE decoder to obtain the generated low-resolution first frame image, and the low-resolution image is input into the SwinIR image super-resolution model to obtain the generated first frame image of the original resolution, which is the first frame image of the generated video.

[0119] (3) Starting from generating the second frame of image, the previous low-resolution image is first input into the VQVAE encoder to obtain the hidden layer features of the previous frame, and then combined with the interactive information operation corresponding to generating the current frame, the current frame image of the original resolution is generated through the Transformer encoder-decoder parameters, VQVAE decoder, and SwinIR image super-resolution model.

[0120] (4) Repeat step (2) until 300 frames of video images are generated. The generated 300 frames of images are integrated into a generated video and output and saved.

[0121] Figure 5 A schematic diagram of a real-time interactive video generation system according to the present application.

[0122] See also Figure 5 It can be seen that this embodiment also provides a real-time interactive video generation system, the generation system comprising:

[0123] Data acquisition module 1, the data acquisition module 1 is configured to acquire training data, the training data includes interaction information and video data corresponding to the interaction information. Specifically, in this embodiment, the data acquisition module 1 is used to execute all logical processes for acquiring training data in the above-mentioned real-time interactive video generation method.

[0124] The data preprocessing module 2 is configured to preprocess the training data to obtain target training data. Specifically, in this embodiment, the data preprocessing module 2 is used to implement all logical processes for preprocessing the training data in the above-mentioned interactive video generation method.

[0125] The model training module 3 is configured to perform model training according to the target training data, and the model training includes a model pre-training process and a video pre-generation process; the model training process includes the training of a low-resolution image reconstruction model and the training of an image super-resolution model; the video pre-generation process includes performing video pre-generation according to an interactive video generation model, so as to complete the training of the interactive video generation model, wherein the interactive video generation model has the trained low-resolution image reconstruction model and the image super-resolution model. Specifically, in the present embodiment, the model training module 3 is used to implement all logical processes for model training in the above-mentioned interactive video generation method.

[0126] The video generation module 4 includes the trained interactive video generation model, and the trained interactive video generation model includes the trained resolution image reconstruction model and the image super-resolution model; the video generation module 4 is configured to generate interactive videos using the trained interactive video generation model. Specifically, in this embodiment, the video generation module 4 is used to implement all logical processes for generating interactive videos in the above-mentioned interactive video generation method.

[0127] This embodiment has the following advantages:

[0128] The real-time interaction information is used as a condition for the generation of low-resolution video frames, and the super-resolution model is used to increase the size of the generated low-resolution frame images to obtain high-resolution and clear video frame images. In this process, the generation of low-resolution video frame images and the process of super-resolution to improve the clarity of video frame images can be carried out in parallel, and the generated video stream can be output in real time.

Claims

1. An interactive video generation method, characterized in that: The method comprises: Acquire training data, wherein the training data includes video data and interaction information corresponding to the video data; the video data is composed of a plurality of frames of low-resolution images; Acquire the image hidden layer features of the initial frame from the training data; Performing feature conversion processing according to the image hidden layer features and the interactive information of the corresponding frame to obtain the first frame image and the target image hidden layer features; Performing image reconstruction processing according to the hidden layer features of the target image to obtain a reconstructed low-resolution image; Performing image super-resolution processing according to the reconstructed low-resolution image to obtain pre-generated image frames, until the pre-generated image frames corresponding to each number of frames in the training data are obtained, thereby obtaining a video generation model; The video generation model is used to process the input data and the target video is obtained.

2. The interactive video generation method according to claim 1, characterized in that: The step of acquiring the image hidden layer features of the initial frame from the training data comprises: Using the training data to train a low-resolution image reconstruction model, wherein the low-resolution image reconstruction model is built into the interactive video generation model; Acquire hidden layer features of each frame of image during the process of training the low-resolution image reconstruction model; Select any one feature from all the hidden layer features as the image hidden layer feature.

3. The interactive video generation method according to claim 2, characterized in that: The step of acquiring the image hidden layer features of the initial frame from the training data also includes: After completing the training of the low-resolution image reconstruction model, obtaining the hidden layer features of each frame of the image; Sampling is performed from the prior distribution of all the hidden layer features to obtain the image hidden layer features.

4. The interactive video generation method according to claim 3, characterized in that: The step of performing feature conversion processing according to the interactive information of the image hidden layer features and the corresponding frame number includes: Acquire the interaction information of the corresponding number of frames according to the hidden layer features of the image; The image hidden layer features and the interactive information of the corresponding frame number are imported into the converter of the interactive video generation model for feature conversion processing to obtain the first frame image and the target image hidden layer features.

5. The interactive video generation method according to claim 4, characterized in that: The step of performing image reconstruction processing according to the hidden layer features of the target image comprises: The target image hidden layer features are imported into the trained low-resolution image reconstruction model, and image reconstruction processing is performed to obtain the reconstructed low-resolution image.

6. The interactive video generation method according to claim 5, characterized in that: The interactive video generation model also has a built-in trained super-resolution model, and the step of performing image super-resolution processing based on the reconstructed low-resolution image includes: The reconstructed low-resolution image is imported into the super-resolution model for super-resolution processing to obtain the pre-generated image frame.

7. The interactive video generation method according to claim 4, characterized in that: The converter includes an interactive information encoder, a conversion encoder and a conversion decoder, and the method further includes: Importing the interactive information corresponding to the low-resolution image into the interactive information encoder for interactive encoding to obtain interactive hidden features, and transmitting the interactive hidden features to the transform decoder through the transform encoder; The image hidden layer features are introduced into the conversion decoder, and the target hidden layer features of the corresponding image frame are generated by the conversion decoder according to the interactive hidden features and the image hidden layer features.

8. The interactive video generation method according to claim 5, characterized in that: The low-resolution image reconstruction model includes an image reconstruction encoder, and the method further includes: The low-resolution image with a corresponding number of frames is imported into the image reconstruction encoder for image reconstruction encoding to obtain the image hidden layer features.

9. The interactive video generation method according to claim 8, characterized in that: The low-resolution image reconstruction model also includes an image reconstruction decoder, and the method further includes: The target hidden layer features are introduced into the image reconstruction decoder for image reconstruction decoding to obtain the reconstructed low-resolution image.

10. The interactive video generation method according to claim 1, characterized in that: The resolution of the pre-generated image frame is the same as the resolution of the video data.

Images