A real-time interactive video generation method and system

By training low-resolution image reconstruction models and super-resolution models, an interactive video generation model is generated, which solves the problem of fixed video content in the prior art, and realizes real-time video stream generation in real-time feedback and highly interactive scenarios. Improves the flexibility and interactivity of video generation.

CN118485578BActive Publication Date: 2024-11-05NANJING SILICON INTELLIGENCE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410939073.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-05
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

Existing video generation technology cannot achieve real-time interactive adjustments, and the generated video content is fixed, which cannot adapt to the needs of immediate feedback and highly interactive scenes.

Method used

By obtaining interactive information and video data for preprocessing, low-resolution image reconstruction model and super-resolution model are trained to generate interactive video generation models, and the model is used for real-time video generation to achieve dynamic adjustment of video content.

Benefits of technology

Real-time video stream generation in scenarios requiring immediate feedback and highly interactive, can dynamically adjust video content according to real-time interactive operations, improving the flexibility and interactivity of video generation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118485578B_ABST
    Figure CN118485578B_ABST
Patent Text Reader

Abstract

The present application provides a real-time interactive video generation method and system, which relates to the field of deep learning technology, including obtaining training data, the training data including interactive information and video data corresponding to the interactive information; preprocessing the training data to obtain target training data; performing model training according to the target training data, the model training including a model pre-training process and a video pre-generation process; the model pre-training process includes training of a low-resolution image reconstruction model and training of an image super-resolution model; the video pre-generation process includes performing video pre-generation according to an interactive video generation model, thereby completing the training of the interactive video generation model, the interactive video generation model having a trained low-resolution image reconstruction model and a super-resolution model; and generating interactive videos using the trained interactive video generation model. This method solves the problem that the existing video generation technology cannot perform real-time interaction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a real-time interactive video generation method and system. Background Art

[0002] With the rapid development of information technology, video generation and processing technology has become a major research hotspot in the field of contemporary science and technology. Among the many video processing technologies, interactive video generation methods are particularly eye-catching. This method can not only realize the basic functions of traditional video production, but also enable users to interact with video content in real time, greatly enriching the scenarios and possibilities of video applications.

[0003] In the current research, the video generation technology mainly includes the following two options. The first option is to import text information into the neural network, generate a frame of video based on the text information through the neural network, and obtain each subsequent frame in the same way to obtain the video image corresponding to the text information; the second option is to obtain the time series noise by importing the time series noise into the 3D generation model to obtain the video image.

[0004] However, whether it is solution 1 or solution 2, the generated video content is fixed, that is, each frame of the image is predetermined and cannot be dynamically adjusted according to real-time interactive operations. Secondly, most existing technologies output in the form of video files and cannot generate real-time video streams, which is particularly insufficient in scenarios that require instant feedback and high interactivity. Summary of the invention

[0005] The present application provides a real-time interactive video generation method and system to solve the problem that existing video generation technology cannot perform real-time interactive adjustment.

[0006] In a first aspect, the present application provides a real-time interactive video generation method, the generation method comprising:

[0007] Acquire training data, where the training data includes interaction information and video data corresponding to the interaction information;

[0008] Preprocessing the training data to obtain target training data;

[0009] Model training is performed according to the target training data, wherein the model training includes a model pre-training process and a video pre-generation process; the model pre-training process includes training of a low-resolution image reconstruction model and training of a super-resolution model; the video pre-generation process includes performing video pre-generation according to an interactive video generation model, thereby completing the training of the interactive video generation model, wherein the interactive video generation model has the trained low-resolution image reconstruction model and the super-resolution model;

[0010] The interactive video is generated using the trained interactive video generation model.

[0011] Preferably, the step of preprocessing the training data includes:

[0012] Downsampling the video data to obtain a low-resolution image, wherein the ratio of reducing the resolution by the downsampling process is determined according to the size of the frame image of the video data;

[0013] The low-resolution image is cropped to obtain a plurality of video segments, each of which includes T frames, and two adjacent video segments are separated by 2 / T frames, and two adjacent video segments overlap by 2 / T frames;

[0014] The corresponding interaction information is recorded for each of the video clips to obtain the target training data, wherein the frame rate of the interaction information recorded in each of the video clips is not less than the number of frames corresponding to the video clip, and the frame rate of the interaction information recorded in each of the video clips is an integer multiple of the number of frames corresponding to the video clip.

[0015] Preferably, recording the corresponding interaction information for each video segment includes performing Gaussian smoothing processing on the corresponding interaction information recorded in the video segment in a time dimension.

[0016] Preferably, the steps of the video pre-generation process further include:

[0017] Generate image hidden layer features corresponding to the initial frame of the low-resolution image;

[0018] Inputting the image hidden layer features and the interactive information of the corresponding low-resolution image into the converter of the interactive video generation model to generate the target hidden layer features corresponding to the first frame image;

[0019] Performing image reconstruction on the target hidden layer features to obtain a reconstructed low-resolution image;

[0020] The reconstructed low-resolution image is subjected to super-resolution processing to obtain a pre-generated first frame image corresponding to the pre-generated video, and each frame image corresponding to the pre-generated video is generated thereby, wherein the resolution of the pre-generated first frame image is the same as the resolution of the video data.

[0021] Preferably, the generating of the image hidden layer features corresponding to the initial frame of the low-resolution image includes a first method and a second method;

[0022] The first method comprises obtaining hidden features of all images during the training of the low-resolution image reconstruction model, and selecting one feature from all hidden features as the image hidden feature;

[0023] The second method includes sampling from the prior distribution of all hidden layer features to obtain the image hidden layer features after completing the training of the low-resolution image reconstruction model.

[0024] Preferably, the steps of the video pre-generation process further include:

[0025] Inputting the target hidden layer features into an image reconstruction decoder, performing image reconstruction on the target hidden layer features through the image reconstruction decoder to obtain the reconstructed low-resolution image, wherein the image reconstruction decoder is a decoder of the trained low-resolution image reconstruction model;

[0026] The reconstructed low-resolution image is input into the trained super-resolution model, and the reconstructed low-resolution image is subjected to super-resolution processing by the super-resolution model to obtain each frame image corresponding to the pre-generated video with the same resolution as the video data.

[0027] Preferably, the steps of the video pre-generation process further include:

[0028] When the t-th frame pre-generated image is generated, the image frame of the t-1 frame is input into the image reconstruction encoder to obtain the image hidden layer features of the t-1 frame image;

[0029] Obtaining the interactive information corresponding to the t-1 frame image, and inputting the image hidden layer features and the interactive information corresponding to the t-1 frame image into the converter of the interactive video generation model to obtain the target hidden layer features corresponding to the t-1 frame image;

[0030] Inputting the target hidden layer features corresponding to the t-1 frame image into an image reconstruction decoder, and reconstructing the target hidden layer features of the t-1 frame image through the image reconstruction decoder to obtain a reconstructed low-resolution image corresponding to the t-1 frame image;

[0031] The reconstructed low-resolution image corresponding to the t-1 frame image frame is input into the trained super-resolution model, and the reconstructed low-resolution image of the t-1 frame image frame is super-resolution processed by the super-resolution model to obtain the t-1 frame pre-generated image frame.

[0032] Preferably, the converter includes an interactive information encoder, a conversion encoder and a conversion decoder, and the steps of the video pre-generation process further include:

[0033] Inputting the interaction information corresponding to the low-resolution image into an interaction information encoder to obtain interaction hidden features corresponding to the interaction information of the low-resolution image;

[0034] The image hidden layer features are input into the conversion decoder, the interactive hidden features are transmitted to the conversion decoder through the conversion encoder, and the target hidden layer features of the corresponding image frame are generated by the conversion decoder according to the interactive hidden features and the image hidden layer features.

[0035] Preferably, the steps of the video pre-generation process further include:

[0036] Merging all generated image frames into the pre-generated video;

[0037] Calculating a reconstruction loss function of the pre-generated video and a real video corresponding to the video data, wherein the reconstruction loss function includes MAE loss, MSE loss, perceptual loss and image similarity loss;

[0038] Constructing a cross entropy loss function, wherein the cross entropy loss function is used to adjust parameters of a video discriminator, wherein the video discriminator is a component in the interactive video generation model used to adjust the definition of the pre-generated video to gradually approach the definition of the real video;

[0039] The parameters of each component in the interactive video generation model are adjusted through the reconstruction loss function and the cross entropy loss function.

[0040] Preferably, the step of adjusting the parameters of each component in the interactive video generation model through the reconstruction loss function and the cross entropy loss function includes calculating the gradient of the parameters of each component in the interactive video generation model through the reconstruction loss function, and adjusting the parameters of each component in the interactive video generation model by gradient descent.

[0041] In a second aspect, the present application further provides a real-time interactive video generation system, the generation system comprising:

[0042] A data acquisition module, wherein the data acquisition module is configured to acquire training data, wherein the training data includes interaction information and video data corresponding to the interaction information;

[0043] A data preprocessing module, wherein the data preprocessing module is configured to preprocess the training data to obtain target training data;

[0044] A model training module, wherein the model training module is configured to perform model training according to the target training data, wherein the model training includes a model pre-training process and a video pre-generation process; the model pre-training process includes training of a low-resolution image reconstruction model and training of a super-resolution model; the video pre-generation process includes performing video pre-generation according to an interactive video generation model, thereby completing the training of the interactive video generation model, wherein the interactive video generation model has the trained low-resolution image reconstruction model and the super-resolution model;

[0045] A video generation module, wherein the video generation module includes the trained interactive video generation model, the trained interactive video generation model includes the trained resolution image reconstruction model and the super-resolution model; the video generation module is configured to generate interactive video using the trained interactive video generation model.

[0046] As can be seen from the above content, the present application provides a real-time interactive video generation method, the generation method includes obtaining training data, the training data includes interactive information and video data corresponding to the interactive information; preprocessing the training data to obtain target training data; performing model training according to the target training data, the model training includes a model pre-training process and a video pre-generation process; the model pre-training process includes training of a low-resolution image reconstruction model and training of a super-resolution model; the video pre-generation process includes pre-generating a video according to an interactive video generation model, thereby completing the training of the interactive video generation model, the interactive video generation model having the trained low-resolution image reconstruction model and the super-resolution model; generating an interactive video using the trained interactive video generation model. The present application solves the problem that existing video generation technology cannot perform real-time interactive adjustment through the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0048] Figure 1 A flowchart of a real-time interactive video generation method for this application;

[0049] Figure 2 A schematic diagram of data processing for training a low-resolution image reconstruction model and a super-resolution model in a real-time interactive video generation method of the present application;

[0050] Figure 3 A schematic diagram of data processing for training an interactive video generation model in a real-time interactive video generation method of the present application;

[0051] Figure 4 A schematic diagram of data processing for training a converter in an interactive video generation model for this application;

[0052] Figure 5 A schematic diagram of a real-time interactive video generation system according to the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] In a specific interactive scenario, the acquired continuous interactive operations are used to generate continuous video images, and the content contained in the generated video images matches the interactive operations. The above interactive scenarios include but are not limited to using keyboard or mouse clicks, game controller operations, voice control, motion capture of somatosensory devices, motion capture using photographic equipment, or brain-computer interfaces. The above video images and content are generated by the above interactions, and the content changes with the changes in the above interactions, including but not limited to game images, scene guides and other video content.

[0055] For example, in a certain game, the keyboard direction keys are used to provide operations, and the game screen generates corresponding images as the keyboard input changes; continuous keyboard input and corresponding image content are collected to train the interactive video generation model; continuous keyboard interactions are input into the interactive video generation model to generate continuous image content, that is, video content; this video content is related to a specific scene, that is, a certain interactive video generation model can generate video content for a specific scene; if video content for a new scene needs to be generated, it is necessary to re-collect data and train a new interactive video generation model.

[0056] The existing technologies for generating videos include the following:

[0057] In the first scheme, the existing video generation process includes (1) inputting text information into the Transformer neural network Decoder model, encoding the text information, and generating the encoding features of the corresponding frame image; (2) using the VQVAE neural network Decoder model to decode the encoding features of the frame image into the current frame image; (3) downsampling the generated current frame image and inputting it into the VQVAE neural network Encoder model to obtain the encoding information of the generated image; (4) inputting the text information and the encoding information of the generated image into the Transformer neural network Decoder model to obtain the corresponding encoding features of the next frame image; (5) repeating steps (2) to (4) until the preset number of frames is reached and the complete video is output.

[0058] The second scheme is to generate a video in the following process: (1) randomly sample temporal noise and input the temporal noise into a 3D generator model with a structure similar to StyleGAN2 and StyleGAN3 through a mapping network. The 3D generator model upsamples the initial input from the temporal dimension and the spatial dimension respectively to generate a low-resolution video; (2) input the low-resolution video generated in step (1) into a video super-resolution model to obtain a high-resolution generated video.

[0059] The third solution uses text information as a condition for video generation, or it can generate videos unconditionally. The third solution uses a diffusion model with 3D-UNet as the backbone to sample noise from a standard Gaussian distribution and gradually denoise it to obtain a generated video. In the gradual denoising process, a language model, such as pre-trained BERT, CLIP, etc., can be used to extract text features from given text information, and the text features can be injected into 3D-UNet using a self-attention mechanism to provide text conditional guidance for video generation. Of course, it is also possible to generate videos unconditionally without using text information.

[0060] According to the above schemes, we can know that: the first scheme is conditional video generation using text information, the second scheme is unconditional video generation without using other information, and the third scheme is both conditional video generation using text information and unconditional video generation without using other information. In the above method, given text information or given initial state, the generated video content is fixed, that is, the image of each frame of the generated video is fixed. Each frame of the generated video is only related to the initial input. If the input changes during the generation process, that is, interaction occurs, the above method cannot handle it. At the same time, the output of the second and third schemes is in the form of video, and it is impossible to generate real-time video streams.

[0061] Based on the above content, this embodiment provides the following solution to solve the above problem.

[0062] Figure 1 This is a flowchart of a real-time interactive video generation method for the present application.

[0063] See also Figure 1 It can be seen that this embodiment provides a real-time interactive video generation method, the generation method comprising:

[0064] S100, obtaining training data, the training data including interaction information and video data corresponding to the interaction information. Specifically, in the present embodiment, the training data is continuous interaction information and video data corresponding to the interaction. Here, the interaction information includes but is not limited to keyboard or mouse clicks, game controller operations, voice control, motion capture of somatosensory devices, motion capture using photographic equipment, or brain-computer interface, etc. These interaction information are directly collected using some external devices; or some external devices are used to collect data first, and then the relevant software or algorithm extracts it from the collected data. The video data corresponding to the interaction refers to the video generated by the above-mentioned interaction process, and the changes in the above-mentioned interaction process can correspond to the changes in the video content. Taking racing games as an example, keyboard operations can control the movement of objects in the game screen, and when the keyboard operation changes, the content of the game screen will change accordingly.

[0065] When acquiring training data, interactive operation information and video image data must be recorded simultaneously. The longer the video used for training, the better, not less than 10 minutes, with no specific restrictions. The video resolution can be a common resolution, such as 360P, 480P, 720P, 1080P, etc., but it is not mandatory. The video frame rate can be the currently commonly used video frame rate, such as 25 frames / second or 30 frames / second, and it is not mandatory. At the same time, record the interactive operation information synchronized with the video, the frame rate of the interactive operation information is not lower than the video frame rate, and the frame rate of the interactive operation information is an integer multiple of the video frame rate. For example, if the recorded video frame rate is 30 frames / second, the frame rate of the recorded interactive operation information can be 30 frames / second, 60 frames / second, 90 frames / second, 120 frames / second, etc.

[0066] S200, preprocessing the training data to obtain target training data. Specifically, in this embodiment, in order to facilitate subsequent model training, the training data needs to be preprocessed first, wherein the step of preprocessing the training data includes:

[0067] The video data is downsampled to obtain a low-resolution image, wherein the ratio of reducing the resolution by the downsampling is determined according to the size of the frame image of the video data; the low-resolution image is cropped to obtain a plurality of groups of video clips, each of the video clips includes T frames, two adjacent video clips are separated by 2 / T frames, and two adjacent video clips overlap by 2 / T frames; the corresponding interaction information is recorded for each of the video clips to obtain the target training data, wherein the frame rate of the interaction information recorded in each of the video clips is not less than the number of frames corresponding to the video clip, and the frame rate of the interaction information recorded in each of the video clips is an integer multiple of the number of frames corresponding to the video clip.

[0068] Exemplary, for training image reconstruction model and super-resolution model , it is necessary to extract all frames of images from the captured video. The resolution of these frames is the same as the resolution of the video, which is recorded as the original resolution here. . For native resolution Perform a certain percentage of downsampling to obtain low resolution The downsampling ratio needs to be determined according to the size of the frame image. For example, if the video frame image is 360P or 480P, the downsampling ratio is 2, which means the lower resolution after downsampling. The length and width are the original resolution of ; If the video frame image is 720P or 1080P, the downsampling ratio is 4, that is, the low resolution after downsampling The length and width are the original resolution of The higher the resolution of the original image, the higher its downsampling ratio.

[0069] It should be noted that the low resolution and native resolution In the form of an image.

[0070] In order to train an interactive video generation model, the original video needs to be cut into several segments, and the image part of each video segment is recorded as , A set of frame images composition, The number of frames contained in each video clip, is the index of the frame image in the video segment. When the video is cut into segments, each segment contains frames, and the interval between adjacent segments is frames, then the adjacent fragments have Frame overlap. Video clip The corresponding interactive operation information is Each frame image There are corresponding interactive operations Since the frame rate of the interactive operation information is not lower than the video frame rate during data recording, and the frame rate of the interactive operation information is an integer multiple of the video frame rate (denoted as ), then each frame of video image corresponds to a set of interactive operations ,in for The index of the interactive operation information corresponding to the frame image, is an interactive information vector. Due to the diversity of interactive forms, It may be discrete or continuous. For example, when the interaction information is collected from keyboard or mouse clicks or game controller operations, the interaction information is expressed as a discrete value of whether the button is clicked or not. is a set of one-hot encoded vectors; when the interaction information is collected by voice control, motion capture of somatosensory devices, motion capture using photographic equipment, or brain-computer interface, the interaction information is represented by a set of continuous values. represent The number of characteristics of a vector is For one In order to unify the discrete interactive operation information and the continuous interactive operation information, in this embodiment, In the time dimension (i.e. ) performs Gaussian smoothing to convert discrete interaction information into continuous interaction information, so that continuous interaction information can be obtained for training and inference under different interaction modes.

[0071] For some more complex interaction methods, such as voice control, motion capture using somatosensory devices, motion capture using photographic equipment, or brain-computer interfaces, the raw data collected by them may include sound signals, images, electrical signals, etc. The process of processing the raw data collected by each complex interaction method is not included in the data preprocessing process of this embodiment. The data preprocessing of the interactive operation information in this embodiment uses the information generated by these interaction methods themselves after processing their raw data. For example, when using voice control as an interaction method, this embodiment does not directly use sound signals, but uses some existing methods to process sound signals, and uses the processed sound features as interactive operation information.

[0072] S300, performing model training according to the target training data, the model training including a model pre-training process and a video pre-generation process; the model pre-training process includes training of a low-resolution image reconstruction model and training of a super-resolution model; the video pre-generation process includes performing video pre-generation according to an interactive video generation model, thereby completing the training of the interactive video generation model, wherein the interactive video generation model has the trained low-resolution image reconstruction model and the super-resolution model. Specifically, in this embodiment, step S300 is a model training step, and the model pre-training process is divided into two stages: (1) a pre-training process, the pre-training process includes training of a low-resolution image reconstruction model and training of a super-resolution model. (2) Training of the interactive video generation model, generating images frame by frame, merging all generated images to generate a video, or outputting images to a video stream in real time. There is no order requirement for the training of the two models in the pre-training process. The pre-training process must be performed before the training of the interactive video generation model.

[0073] Figure 2 A schematic diagram of data processing for training a low-resolution image reconstruction model and a super-resolution model in a real-time interactive video generation method of the present application.

[0074] See also Figure 2 It can be seen that training a low-resolution image reconstruction model includes the following schemes.

[0075] By image encoder and image decoder It is composed of convolutional layers, fully connected layers, activation layers, pooling layers, normalization layers and other basic neural network structures. Some existing network structures can be used, such as AutoEncoder, VAE, VQVAE and other neural networks with encoder and decoder structures, or the basic structure can be used to design the image reconstruction model. There is no requirement for the specific structure of the neural network here.

[0076] Low-resolution image reconstruction model uses low-resolution As input, the image encoder , get the input low resolution The hidden layer code , the hidden layer encoding Input image decoder , obtain the reconstructed low-resolution , hoping to reconstruct the image With the input image As much as possible. In this process, the image encoder The resulting hidden code It can be used to represent input images and subsequently used for training and inference of interactive video generation models.

[0077] To train the low-resolution image reconstruction model, use the real low-resolution As input, and also as supervision data. After the image encoder and image decoder After that, the model outputs the reconstructed low-resolution , calculate the reconstruction loss function The reconstruction loss function is based on the MAE loss (L 1 loss), MSE loss (L 2 loss), perceptual loss, image similarity loss, or any other loss function that can characterize image differences. After calculating the reconstruction loss function, the back propagation process is performed. According to the calculated loss function, the image encoder and image decoder The parameters of each component in the low-resolution image reconstruction model are calculated using the gradient descent method to optimize and update the parameters of each component in the low-resolution image reconstruction model. The model pre-training process includes several rounds, and each round executes the above model pre-training process until the training termination condition is met and the model training is completed.

[0078] It should be noted that when using different neural networks, the loss functions that need to be calculated may be different.

[0079] See also Figure 2 It can be seen that training super-resolution model The following solutions are included.

[0080] Super-resolution model The input low-resolution image can be processed to obtain a high-resolution image corresponding to the content of the low-resolution image to improve the clarity of the input low-resolution image. Super-resolution model It is composed of basic neural network structures such as convolutional layer, fully connected layer, activation layer, pooling layer, normalization layer, etc. Existing neural networks for super-resolution tasks can be used as super-resolution models. , you can also use the basic structure to design your own super-resolution model There is no requirement for the specific structure of the neural network. Including but not limited to HAT, SwinIR, LTE, etc.

[0081] Super-resolution model It contains a series of upsampling layers, using low-resolution As input, after upsampling, the output is the original resolution of the reconstruction , hoping to reconstruct the image With true native resolution As identical as possible.

[0082] To train the super-resolution model , using a realistic low-resolution As input, use its corresponding native resolution As supervised data. After super-resolution model After that, the reconstructed original resolution is output , calculate the reconstruction loss The reconstruction loss function can use MAE loss (L 1 loss), MSE loss (L 2 loss), perceptual loss, image similarity loss, or any other loss function that can characterize image differences. Optimize the reconstruction loss function to ensure the super-resolution model The output reconstructed image has the same content as the true original resolution image.

[0083] It should be noted that the original resolution of the reconstruction In the form of an image.

[0084] By optimizing the super-resolution discriminator , to determine whether the input image is a real image or a reconstructed image, so that the super-resolution model The clarity of the output reconstructed image is close to the real image. Super-resolution discriminator Can optimize super-resolution models , so that the super-resolution model The output image is closer to the real image. Super-resolution discriminator It is composed of basic neural network structures such as convolutional layer, fully connected layer, activation layer, pooling layer, normalization layer, etc. You can use the existing discriminator structure of the generative adversarial network, or you can use the basic structure to design your own discriminator.

[0085] Super-resolution discriminator It is used to determine whether the input image is a real original resolution image or a reconstructed original resolution image. It is a model for binary classification. Therefore, the commonly used cross entropy loss function can be used. Cross Entropy Loss Function As shown in formula (1), Input super-resolution discriminator images, The label for this image. When input to the super-resolution discriminator Image For a true original resolution image, is 1; when the input super-resolution discriminator When the image is the reconstructed original resolution image, is 0. Feed the image to the super-resolution discriminator The output after , whose value represents the probability that the input image is a real image. In addition to using the cross entropy loss function for discriminator classification, other loss functions for classification can also be used.

[0086] (1)

[0087] Calculate the above reconstruction loss and the cross entropy loss of the discriminator After that, the back propagation process is performed. According to the calculated loss function, the gradient of the parameters of each component in the model is calculated, and the gradient descent method is used to optimize and update the parameters of each component in the model. It should be noted that the above reconstruction loss Only used to optimize super-resolution models Parameters, discriminator loss Also used to optimize super-resolution models Parameters and super-resolution discriminator Parameters. The model pre-training process includes several rounds, and each round executes the above model pre-training process until the training termination condition is met and the model training is completed.

[0088] Figure 3 A schematic diagram of data processing for training an interactive video generation model in a real-time interactive video generation method of the present application.

[0089] See also Figure 3 It can be seen that the scheme for training an interactive video generation model is as follows.

[0090] Among them, the video pre-generation process (i.e., training an interactive video generation model) includes generating image hidden layer features corresponding to the initial frame of the low-resolution image; inputting the image hidden layer features and the interactive information of the corresponding low-resolution image into the converter of the interactive video generation model to generate target hidden layer features corresponding to the first frame image; reconstructing the target hidden layer features to obtain a reconstructed low-resolution image; performing super-resolution processing on the reconstructed low-resolution image to obtain a pre-generated first frame image corresponding to the pre-generated video, and thereby generating each frame image corresponding to the pre-generated video, and the resolution of the pre-generated first frame image is the same as the resolution of the video data.

[0091] in, Figure 3 Transformer encoder-decoder Interactive Encoder See as a converter in the above content.

[0092] The interactive video generation model contains components such as interactive information encoder and Transformer encoder-decoder. , Image Encoder , Image Decoder and super-resolution models . The image encoder , Image Decoder and super-resolution models This has been obtained by the above pre-training process. Therefore, the process of training an interactive video generation model only requires training the interactive information encoder and the Transformer encoder-decoder .

[0093] Figure 4 Schematic diagram of the data processing for the converter in the interactive video generation model trained for this application.

[0094] See also Figure 4 It can be seen that, further, in some embodiments, the converter includes an interactive information encoder, a conversion encoder and a conversion decoder. Therefore, the step of generating the target hidden layer features corresponding to the first frame image also includes inputting the interactive information corresponding to the low-resolution image into the interactive information encoder to obtain interactive hidden features corresponding to the interactive information of the low-resolution image; inputting the image hidden layer features into the conversion decoder, transmitting the interactive hidden features to the conversion decoder through the conversion encoder, and generating the target hidden layer features of the corresponding image frame through the conversion decoder according to the interactive hidden features and the image hidden layer features.

[0095] In order to avoid confusion, the text description will be Figure 4 Transformer encoder in It is called the conversion encoder, and the Transformer decoder It is called a transform decoder.

[0096] Exemplary, where the mutual information encoder Use pre-processed interaction information As input, the network is fed forward and nonlinearly transformed to obtain the hidden features of the interactive operation information. It is composed of basic neural network structures such as convolutional layer, fully connected layer, activation layer, pooling layer, and normalization layer. Figure 3 and Figure 4 In the interactive operation information encoder The input is interactive operation information , that is, the video to be generated The interactive operation information corresponding to the frame. In fact, according to different video generation tasks, the interactive information encoder The input can also be in various forms.

[0097] The above different video tasks are divided into: (1) Offline tasks. Given a set of known interactive operation information ,in is the number of frames of the video to be generated, To generate the video Frame. Use known interaction information , directly generate the corresponding video . (2) Real-time task. Given the interactive information of the historical frame and the current frame , generating the current time The corresponding video frame , the generated video frames are output in the form of video streams to achieve real-time video generation.

[0098] For offline tasks, since the interactive operation information corresponding to all frames is known, the interactive information encoder You can use only The interactive operation information corresponding to the frame can also be used The frame and several frames before and after it correspond to interactive information, such as When using multiple frames of interactive operation information, it is only necessary to splice the interactive operation information of multiple frames in the time dimension.

[0099] For real-time tasks, since only the interactive operation information corresponding to the historical frame and the current frame is known, the interactive information encoder You can use only The interactive operation information corresponding to the frame can also be used The interactive operation information corresponding to the frame and its previous frames, such as When using multiple frames of interactive operation information, it is only necessary to splice the interactive operation information of multiple frames in the time dimension.

[0100] Transformer Encoder-Decoder The structure is the commonly used Transformer model, which will not be explained in detail here. Transformer encoder Encode the hidden features of the input interactive operation information, and use its internal self-attention mechanism to calculate the importance of each step of the sequential interactive operation information to the generation of the corresponding frame. Transformer decoder Using the Encoder The output of is used as input, and the hidden features of the image generated by the previous frame are used As input. The generated previous frame image is input into the image encoder You can get Transformer Decoder The role of is to combine the interactive operation information with the information of the image generated in the previous frame to generate the hidden layer features of the current frame , used for image decoder Generate current frame image , and then through the super-resolution model Improve the clarity of the image and finally obtain the image generation frame of the current frame .

[0101] Furthermore, after completing the generation of a video, the present embodiment further optimizes the scheme, and the specific scheme is as follows: merging all generated image frames into the pre-generated video; calculating the reconstruction loss function of the pre-generated video and the real video corresponding to the video data, the reconstruction loss function including MAE loss, MSE loss, perceptual loss and image similarity loss; constructing a cross-entropy loss function, the cross-entropy loss function is used to adjust the parameters of the video discriminator, the video discriminator is a component in the interactive video generation model used to adjust the clarity of the pre-generated video to gradually approach the real video; adjusting the parameters of each component in the interactive video generation model through the reconstruction loss function and the cross-entropy loss function; the step of adjusting the parameters of each component in the interactive video generation model through the reconstruction loss function and the cross-entropy loss function includes calculating the gradient of the parameters of each component in the interactive video generation model through the reconstruction loss function, and adjusting the parameters of each component in the interactive video generation model by gradient descent.

[0102] Specifically, in this embodiment, the interactive information encoder for training the interactive video generation model , Transformer encoder-decoder , using the preprocessed interaction information as input and the real video clips corresponding to the interaction As supervision information. After the interactive video generation model, the generated video with the same resolution as the real video clip is output , calculate the reconstruction loss of the generated video and the real video The reconstruction loss function is based on the MAE loss (L 1 loss), MSE loss (L 2 Loss), perceptual loss, image similarity loss, or any other loss function that can characterize image differences. Optimize the reconstruction loss function to ensure that the video generated by the interactive video generation model has the same content as the real video.

[0103] Using video discriminator Determine whether the input video is a real video or a generated video, so that the clarity of the video output by the interactive video generation model is close to the real video. Its principle and loss function calculation formula are similar to the super-resolution discriminator Same, no more repetition. Video Identifier With super-resolution discriminator The difference is that the super-resolution discriminator The input is a 2D image, video discriminator The input is 3D video. Video Identifier It is used to judge whether the input video is a real video or a generated video. It is a model for binary classification, so the cross entropy loss function in the same form as formula (1) can also be used. To optimize the video discriminator .

[0104] Calculate the above reconstruction loss and the cross entropy loss of the discriminator After that, the back propagation process is performed. According to the calculated loss function, the gradient of the parameters of each component in the interactive video generation model is calculated, and the gradient descent method is used to optimize and update the parameters of each component in the model. It should be noted that the above reconstruction loss Only used to optimize the mutual information encoder and Transformer encoder-decoder , the discriminator loss function It is also used to optimize the mutual information encoder , Transformer encoder-decoder and video discriminator Parameters. Image encoder , Image Decoder and super-resolution models The parameters of are not updated during this training process. The model pre-training process includes several rounds, and each round executes the above model pre-training process until the training termination condition is met, completing the training of the interactive video generation model.

[0105] Among them, see Figure 3 It can be known that the specific scheme of performing image reconstruction on the target hidden layer features to obtain the reconstructed low-resolution image is to input the target hidden layer features into an image reconstruction decoder, and perform image reconstruction on the target hidden layer features through the image reconstruction decoder to obtain the reconstructed low-resolution image. The image reconstruction decoder is a decoder of the low-resolution image reconstruction model that has completed training.

[0106] Among them, see Figure 3It can be seen that the specific solution of performing super-resolution processing on the reconstructed low-resolution image to obtain the pre-generated first frame image corresponding to the pre-generated video is to input the reconstructed low-resolution image into the super-resolution model trained In the super-resolution model The reconstructed low-resolution image is subjected to super-resolution processing to obtain each frame image corresponding to the pre-generated video having the same resolution as the video data.

[0107] It should be noted that in order to reflect that the interactive video generation model has the trained low-resolution image reconstruction model, Figure 3 Image encoder in and image decoder Called image reconstruction encoder and image reconstruction decoder.

[0108] It should be noted that after the first pre-generated image frame is generated, the generation of subsequent image frames depends on the data of the previous image frame. The specific scheme is as follows: when the t-th pre-generated image is generated, the image frame of the t-1 frame is input into the image reconstruction encoder to obtain the image hidden layer features of the t-1 frame image frame; the interaction information corresponding to the t-1 frame image frame is obtained, and the image hidden layer features and interaction information corresponding to the t-1 frame image frame are input into the converter of the interactive video generation model to obtain the target hidden layer features of the t-1 frame image frame; the target hidden layer features of the t-1 frame image frame are input into the image reconstruction decoder, and the target hidden layer features of the t-1 frame image frame are reconstructed through the image reconstruction decoder to obtain the reconstructed low-resolution image of the t-1 frame image frame; the reconstructed low-resolution image corresponding to the t-1 frame image frame is input into the super-resolution model to complete the training. In the super-resolution model The reconstructed low-resolution image of the t-1 frame image frame is subjected to super-resolution processing to obtain the t-1 frame pre-generated image frame.

[0109] Step S300 can be summarized as follows:

[0110] (1) Generate hidden features of the initial frame There are two generation methods: ① In the process of pre-training the image reconstruction model, the hidden features of all known images are obtained, and the hidden features of an image are randomly selected as ; ② After pre-training the image reconstruction model, the prior distribution of the hidden layer features is known, and samples can be sampled from the prior distribution of the hidden layer features .

[0111] (2) The hidden features of the initial frame And the interactive operation information corresponding to the first frame The hidden layer features are input into the Transformer encoder-decoder , obtain the hidden layer features used to reconstruct the first frame image ;Will Input image decoder , obtain the reconstructed low-resolution image ;Will Input Super Resolution Model , get the generated original resolution image generation frame , that is, the generation of the first frame of the video is completed.

[0112] (3) For generating video Frame image, first generate the previous frame of video image Input image encoder , get the previous frame image The hidden layer code ;Will And the interactive operation information corresponding to the current frame The hidden layer features are input into the Transformer encoder-decoder , obtain the hidden layer features used to reconstruct the current frame image ;Will Input image decoder , obtain the reconstructed low-resolution image ;Will Input Super Resolution Model , get the generated original resolution image generation frame , which completes the video Frame generation.

[0113] (4) Repeat step (3) until the termination condition is reached. Different video generation tasks have different termination conditions. For offline tasks, the termination condition is generally that the number of generated video frames reaches a preset value. At this time, all generated frames are merged into a video For real-time tasks, the generated video frames are output to the video stream, and the termination condition is generally that the user terminates the generation process. The termination condition during the training process is that the number of generated video frames reaches the number of frames in the training video clip.

[0114] Exemplarily, this embodiment provides the following model training example:

[0115] (1) Use the keyboard to operate a game, and record the keyboard operation while recording the game video. The video length is 30 minutes, the video resolution is 512×512 pixels, the video frame rate is 30 frames / second, and the keyboard operation recording frame rate is 300 frames / second, that is, each frame of the video image corresponds to 10 frames of keyboard operation recording.

[0116] (2) Downsample the original video with a resolution of 512×512 pixels to a resolution of 256×256 pixels; extract all frame images from the original video with a resolution of 512×512 pixels, and extract all frame images from the downsampled video with a resolution of 256×256 pixels; crop the 30-minute full video into several video clips, each video clip contains 16 frames of images, and the cropping interval between two adjacent video clips is 8 frames. At the same time, crop the keyboard interaction record data, each video clip contains 160 frames of interaction data, and each frame of image corresponds to 10 frames of interaction data; perform Gaussian smoothing with a mean of 0 and a variance of 3 on the interaction data corresponding to each frame.

[0117] (3) Use the downsampled frame image with a resolution of 256×256 pixels as input data and supervision data to train the image reconstruction model VQVAE to obtain the image encoder and image decoder , and obtain the prior distribution of the image hidden layer features at the same time;

[0118] (4) Use the downsampled frame image with a resolution of 256×256 pixels as input data and the original frame image with a resolution of 512×512 pixels as supervision data to train the SwinIR super-resolution model. ;

[0119] (5) Use the VQVAE encoder to extract hidden features from the previous frame of the generated image (the hidden features of the initial frame are sampled from the prior distribution of the image hidden features obtained by pre-training VQVAE); use the interactive operation information of length 10 corresponding to the current frame to input the interactive information encoder to obtain the interactive information hidden features; input the generated previous frame image hidden features and the interactive information hidden features into the Transformer encoder-decoder , generate the hidden features of the current frame image; input it into the image decoder , generate a low-resolution image of the current frame, that is, its resolution is 256×256 pixels; then input the low-resolution image into the pre-trained SwinIR super-resolution model , and obtain an image with an original resolution of 512×512 pixels, which is the generated current frame image.

[0120] (6) Repeat process (5) 16 times to obtain the generated video clips; use the generated video clips and the real video clips to calculate the relevant loss function to optimize the interactive information encoder and Transformer encoder-decoder Parameters are added until the training is terminated, and a trained interactive video generation model is obtained.

[0121] S400, generating an interactive video using the trained interactive video generation model. Specifically, in this embodiment, during the process of generating the interactive video, the data processing process of the interactive video generation model is as follows:

[0122] ① The interactive information encoder and Transformer encoder-decoder can be The operation process is placed in a process, and its input queue uses the output queue of process ④ to establish an output queue for it;

[0123] ② Image decoder The operation process is placed in a process, and its input queue uses the output queue of process ① to establish an output queue for it;

[0124] ③ Super-resolution model The operation process is placed in a process, and its input queue uses the output queue of process ② to establish an output queue for it;

[0125] ④ Image encoder The operation process is placed in a process, and its input queue uses the output queue of process ② to establish an output queue for it;

[0126] ⑤The final video output process is placed in a process, whose input queue uses the output queue of process ③, and each frame of the generated image is output to a video file or video stream.

[0127] The real-time performance of the model inference process can be enhanced by running in parallel.

[0128] Exemplarily, this embodiment provides the following example of generating an interactive video:

[0129] (1) Given a set of interaction operation sequences of known length, whose length is 3000 frames, the corresponding generated video frame number is 300 frames and the video length is 10 seconds.

[0130] (2) Sample the hidden features of the initial frame from the prior distribution of the image hidden features obtained by the pre-trained VQVAE, and input the hidden features of the initial frame and the interactive operation information corresponding to the generation of the first frame of the image into the Transformer encoder-decoder Parameters, obtain the hidden layer features of the first frame image, then input it into the VQVAE decoder to obtain the generated low-resolution first frame image, and input the low-resolution image into the SwinIR super-resolution model The first frame image of the generated original resolution is obtained, that is, the first frame image of the generated video.

[0131] (3) Starting from generating the second frame of image, the previous low-resolution image is first input into the VQVAE encoder to obtain the hidden layer features of the previous frame of image, and then combined with the interactive information operation corresponding to the generation of the current frame of image, and then passed through the Transformer encoder-decoder in sequence. Parameters, VQVAE decoder, SwinIR super-resolution model Generates the current frame image at original resolution.

[0132] (4) Repeat step (2) until 300 frames of video images are generated. The generated 300 frames of images are integrated into the generated video and output and saved.

[0133] Figure 5 A schematic diagram of a real-time interactive video generation system according to the present application.

[0134] See also Figure 5 It can be seen that this embodiment also provides a real-time interactive video generation system, the generation system comprising:

[0135] Data acquisition module 1, the data acquisition module 1 is configured to acquire training data, the training data includes interaction information and video data corresponding to the interaction information. Specifically, in this embodiment, the data acquisition module 1 is used to execute all logical processes for acquiring training data in the above-mentioned real-time interactive video generation method.

[0136] The data preprocessing module 2 is configured to preprocess the training data to obtain target training data. Specifically, in this embodiment, the data preprocessing module 2 is used to implement all logical processes for preprocessing the training data in the above-mentioned interactive video generation method.

[0137] The model training module 3 is configured to perform model training according to the target training data, and the model training includes a model pre-training process and a video pre-generation process; the model pre-training process includes the training of a low-resolution image reconstruction model and the training of a super-resolution model; the video pre-generation process includes performing video pre-generation according to an interactive video generation model, so as to complete the training of the interactive video generation model, wherein the interactive video generation model has the trained low-resolution image reconstruction model and the super-resolution model. Specifically, in the present embodiment, the model training module 3 is used to implement all logical processes for model training in the above-mentioned interactive video generation method.

[0138] The video generation module 4 includes the trained interactive video generation model, and the trained interactive video generation model includes the trained resolution image reconstruction model and the super-resolution model; the video generation module 4 is configured to generate interactive videos using the trained interactive video generation model. Specifically, in this embodiment, the video generation module 4 is used to implement all logical processes for generating interactive videos in the above-mentioned interactive video generation method.

[0139] This embodiment has the following advantages:

[0140] The real-time interaction information is used as a condition for the generation of low-resolution video frames, and the super-resolution model is used to increase the size of the generated low-resolution frame images to obtain high-resolution and clear video frame images. In this process, the generation of low-resolution video frame images and the process of super-resolution to improve the clarity of video frame images can be carried out in parallel, and the generated video stream can be output in real time.

Claims

1. A real-time interactive video generation method, It is characterized in that The generation method comprises: Acquire training data, where the training data includes interaction information and video data corresponding to the interaction information; Preprocessing the training data to obtain target training data; Model training is performed according to the target training data, wherein the model training includes a model pre-training process and a video pre-generation process; the model pre-training process includes training of a low-resolution image reconstruction model and training of a super-resolution model; the video pre-generation process includes performing video pre-generation according to an interactive video generation model, thereby completing the training of the interactive video generation model, wherein the interactive video generation model has the trained low-resolution image reconstruction model and the super-resolution model; Generate an interactive video using the trained interactive video generation model; The steps of the video pre-generation process include: Generate image hidden layer features corresponding to the initial frame of the low-resolution image; Inputting the image hidden layer features and the interactive information of the corresponding low-resolution image into the converter of the interactive video generation model to generate the target hidden layer features corresponding to the first frame image; Inputting the target hidden layer features into an image reconstruction decoder, performing image reconstruction on the target hidden layer features through the image reconstruction decoder to obtain the reconstructed low-resolution image, wherein the image reconstruction decoder is a decoder of the trained low-resolution image reconstruction model; The reconstructed low-resolution image is input into the trained super-resolution model, and the reconstructed low-resolution image is subjected to super-resolution processing by the super-resolution model to obtain each frame image corresponding to the pre-generated video having the same resolution as the video data.

2. A real-time interactive video generation method according to claim 1, It is characterized in that The step of preprocessing the training data comprises: Downsampling the video data to obtain a low-resolution image, wherein the ratio of reducing the resolution by the downsampling process is determined according to the size of the frame image of the video data; The low-resolution image is cropped to obtain a plurality of video segments, each of which includes T frames, and two adjacent video segments are separated by 2 / T frames, and two adjacent video segments overlap by 2 / T frames; The corresponding interaction information is recorded for each of the video clips to obtain the target training data, wherein the frame rate of the interaction information recorded in each of the video clips is not less than the number of frames corresponding to the video clip, and the frame rate of the interaction information recorded in each of the video clips is an integer multiple of the number of frames corresponding to the video clip.

3. A real-time interactive video generation method according to claim 2, It is characterized in that The recording of the corresponding interaction information for each video segment includes performing Gaussian smoothing processing on the corresponding interaction information recorded in the video segment in a time dimension.

4. A real-time interactive video generation method according to claim 1, It is characterized in that The generating of the image hidden layer features corresponding to the initial frame of the low-resolution image includes a first method and a second method; The first method comprises obtaining hidden features of all images during the training of the low-resolution image reconstruction model, and selecting one feature from all hidden features as the image hidden feature; The second method includes sampling from the prior distribution of all hidden layer features to obtain the image hidden layer features after completing the training of the low-resolution image reconstruction model.

5. A real-time interactive video generation method according to claim 1, It is characterized in that The steps of the video pre-generation process also include: When the t-th frame pre-generated image is generated, the image frame of the t-1 frame is input into the image reconstruction encoder to obtain the image hidden layer features of the t-1 frame image; Obtaining the interactive information corresponding to the t-1 frame image, and inputting the image hidden layer features and the interactive information corresponding to the t-1 frame image into the converter of the interactive video generation model to obtain the target hidden layer features corresponding to the t-1 frame image; Inputting the target hidden layer features corresponding to the t-1 frame image into an image reconstruction decoder, and reconstructing the target hidden layer features of the t-1 frame image through the image reconstruction decoder to obtain a reconstructed low-resolution image corresponding to the t-1 frame image; The reconstructed low-resolution image corresponding to the t-1 frame image frame is input into the trained super-resolution model, and the reconstructed low-resolution image of the t-1 frame image frame is super-resolution processed by the super-resolution model to obtain the t-1 frame pre-generated image frame.

6. A real-time interactive video generation method according to claim 1, It is characterized in that The converter includes an interactive information encoder, a conversion encoder and a conversion decoder, and the steps of the video pre-generation process also include: Inputting the interaction information corresponding to the low-resolution image into an interaction information encoder to obtain interaction hidden features corresponding to the interaction information of the low-resolution image; The image hidden layer features are input into the conversion decoder, the interactive hidden features are transmitted to the conversion decoder through the conversion encoder, and the target hidden layer features of the corresponding image frame are generated by the conversion decoder according to the interactive hidden features and the image hidden layer features.

7. A real-time interactive video generation method according to claim 1, It is characterized in that The steps of the video pre-generation process also include: Merging all generated image frames into the pre-generated video; Calculating a reconstruction loss function of the pre-generated video and a real video corresponding to the video data, wherein the reconstruction loss function includes MAE loss, MSE loss, perceptual loss and image similarity loss; Constructing a cross entropy loss function, wherein the cross entropy loss function is used to adjust parameters of a video discriminator, wherein the video discriminator is a component in the interactive video generation model used to adjust the definition of the pre-generated video to gradually approach the definition of the real video; The parameters of each component in the interactive video generation model are adjusted through the reconstruction loss function and the cross entropy loss function.

8. A real-time interactive video generation method according to claim 7, It is characterized in that The step of adjusting the parameters of each component in the interactive video generation model through the reconstruction loss function and the cross entropy loss function includes calculating the gradient of the parameters of each component in the interactive video generation model through the reconstruction loss function, and adjusting the parameters of each component in the interactive video generation model by gradient descent.

9. A real-time interactive video generation system, It is characterized in that The generation system is applicable to the real-time interactive video generation method according to any one of claims 1 to 8, and the generation system comprises: A data acquisition module (1), the data acquisition module (1) being configured to acquire training data, the training data comprising interaction information and video data corresponding to the interaction information; A data preprocessing module (2), wherein the data preprocessing module (2) is configured to preprocess the training data to obtain target training data; A model training module (3), wherein the model training module (3) is configured to perform model training according to the target training data, wherein the model training includes a model pre-training process and a video pre-generation process; the model pre-training process includes training of a low-resolution image reconstruction model and training of a super-resolution model; the video pre-generation process includes performing video pre-generation according to an interactive video generation model, thereby completing the training of the interactive video generation model, wherein the interactive video generation model includes the trained low-resolution image reconstruction model and the super-resolution model; A video generation module (4), wherein the video generation module (4) includes the trained interactive video generation model, wherein the trained interactive video generation model includes the trained high-resolution image reconstruction model and the super-resolution model; the video generation module (4) is configured to generate an interactive video using the trained interactive video generation model; The model training module (3) is also configured as follows: Generate image hidden layer features corresponding to the initial frame of the low-resolution image; Inputting the image hidden layer features and the interactive information of the corresponding low-resolution image into the converter of the interactive video generation model to generate the target hidden layer features corresponding to the first frame image; Inputting the target hidden layer features into an image reconstruction decoder, performing image reconstruction on the target hidden layer features through the image reconstruction decoder to obtain the reconstructed low-resolution image, wherein the image reconstruction decoder is a decoder of the trained low-resolution image reconstruction model; The reconstructed low-resolution image is input into the trained super-resolution model, and the reconstructed low-resolution image is subjected to super-resolution processing by the super-resolution model to obtain each frame image corresponding to the pre-generated video having the same resolution as the video data.