Method and system for generating real-time interactive video

Abstract

The present disclosure relates to a method and system for generating a real-time interactive video in the field of deep learning technology.SOLUTION: The method includes obtaining training data, the training data including interactive information and video data corresponding to the interactive information, preprocessing the training data to obtain target training data, performing model training based on the target training data to complete training of an interactive video generation model, and performing interactive video generation by using the trained interactive video generation model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to the field of deep learning technology, and in particular to a method and system for real-time interactive video generation. [Background technology]

[0002] With the rapid development of information technology, video generation and processing technology has become a major research hot topic in the current field of science and technology. Among many video processing technologies, interactive video generation methods have attracted particular attention. Such methods can not only realize the basic functions of traditional video production, but also give users the ability to interact with video content in real time, greatly enriching the scenes and possibilities of video applications.

[0003] In current research, there are two main technical proposals for video generation technology: Technical proposal 1 involves introducing text information into a neural network, which then generates a frame of video based on the text information, and then obtaining each subsequent frame in a similar manner to obtain a video image corresponding to the text information; Technical proposal 2 involves acquiring temporal noise and introducing this temporal noise into a 3D generative model to obtain a video image.

[0004] However, in both Technical Scheme 1 and Technical Scheme 2, the generated video content is fixed, i.e., the images of each frame are predetermined and cannot be dynamically adjusted based on real-time interactive operations. In addition, the prior art often outputs in the form of a video file, which does not allow for the generation of a real-time video stream, which is particularly insufficient in scenes requiring immediate feedback and high interactivity. Summary of the Invention [Problem to be solved by the invention]

[0005] SUMMARY OF THE INVENTION The present application provides a method and system for real-time interactive video generation to solve the problem that conventional video generation technologies cannot perform interactive adjustments in real time. [Means for solving the problem]

[0006] A real-time interactive video generation method according to a first aspect of the present application comprises: obtaining training data including interactive information and video data corresponding to the interactive information; pre-processing the training data to obtain target training data; a step of completing training for the interactive video generation model by performing model training based on the target training data, the model training including: a model pre-training process including training a low-resolution image reconstruction model and training a super-resolution model; and a video pre-generation process including performing video pre-generation based on the interactive video generation model, wherein the interactive video generation model includes the low-resolution image reconstruction model and the super-resolution model that have been trained; and performing interactive video generation using the interactable video generation model after training has been completed.

[0007] Preferably, the step of pre-processing the training data comprises: performing a downsampling process on the video data to obtain a low resolution image; performing a cropping process on the low-resolution image to obtain a set of video segments; and recording the interactive information corresponding to each of the video segments to obtain the target training data; a rate of resolution reduction due to the downsampling process is determined by a size of a frame image of the video data; Each of the video segments includes T frames, and two adjacent video segments are spaced apart by 2 / T frames, and the two adjacent video segments have an overlap of 2 / T frames; The frame rate of the interactive information recorded in each of the video segments is equal to or greater than the number of frames corresponding to the video segment, and the frame rate of the interactive information recorded in each of the video segments is an integer multiple of the number of frames corresponding to the video segment.

[0008] Preferably, the step of recording the interactive information corresponding to each of the video segments comprises: and performing a Gaussian smoothing process in the time dimension on the corresponding interactive information recorded in the video segment.

[0009] Preferably, the steps of the video pre-generation process include: generating image hidden layer features corresponding to an initial frame of the low-resolution image; inputting the image hidden layer features and interactive information corresponding to the low-resolution image into a transformer of the interactive video generation model to generate target hidden layer features corresponding to a first frame image; performing image reconstruction on the target hidden layer features to obtain a reconstructed low-resolution image; The method further includes a step of performing super-resolution processing on the reconstructed low-resolution image to obtain a pre-generated first frame image corresponding to the pre-generated video, thereby generating each frame image corresponding to the pre-generated video, wherein the resolution of the pre-generated first frame image is the same as the resolution of the video data.

[0010] Preferably, the step of generating image hidden layer features corresponding to the initial frame of the low-resolution image includes a first method and a second method, The first method includes a step of acquiring hidden layer features of all images in a process of training the low-resolution image reconstruction model, and arbitrarily selecting one feature from all the hidden layer features as the image hidden layer feature; The second method includes a step of obtaining the image hidden layer features by sampling from a prior distribution of all hidden layer features after completing training for the low-resolution image reconstruction model.

[0011] Preferably, the steps of the video pre-generation process include: inputting the target hidden layer features into an image reconstruction decoder, and performing image reconstruction on the target hidden layer features by the image reconstruction decoder to obtain the reconstructed low-resolution image, wherein the image reconstruction decoder is a decoder of the low-resolution image reconstruction model whose training has been completed; The method further includes inputting the reconstructed low-resolution image into the super-resolution model that has completed training, and performing super-resolution processing on the reconstructed low-resolution image using the super-resolution model to obtain each frame image corresponding to the pre-generated video having the same resolution as the video data.

[0012] Preferably, the steps of the video pre-generation process include: When generating a t-th frame pre-generated image, input the t-1-th frame image frame into an image reconstruction encoder to obtain image hidden layer features of the t-1-th frame image frame; Obtaining interactive information corresponding to the image frame of the t-1th frame, inputting the image hidden layer features and interactive information corresponding to the image frame of the t-1th frame into a transformer of the interactive video generation model, and obtaining target hidden layer features corresponding to the image frame of the t-1th frame; inputting the target hidden layer features corresponding to the image frame of the t-1th frame into an image reconstruction decoder, and performing image reconstruction on the target hidden layer features of the image frame of the t-1th frame by the image reconstruction decoder to obtain a reconstructed low-resolution image corresponding to the image frame of the t-1th frame; The method further includes inputting a reconstructed low-resolution image corresponding to the image frame of the t-1th frame into the super-resolution model after training, and performing super-resolution processing on the reconstructed low-resolution image of the image frame of the t-1th frame using the super-resolution model to obtain a pre-generated image frame of the t-1th frame.

[0013] Preferably, the transformer comprises an interactive information encoder, a transformer encoder and a transformer decoder; The steps of the video pre-generation process are: inputting the interactive information corresponding to the low-resolution image into an interactive information encoder to obtain interactive hidden features corresponding to the interactive information of the low-resolution image; The method further includes inputting the image hidden layer features to the Transformer decoder, transmitting the interactive hidden features to the Transformer decoder by the Transformer encoder, and generating target hidden layer features corresponding to the image frame based on the interactive hidden features and the image hidden layer features.

[0014] Preferably, the steps of the video pre-generation process include: merging all generated image frames into said pre-generated video; calculating a reconstruction loss function of the pre-generated video and an actual video corresponding to the video data, wherein the reconstruction loss function comprises an MAE loss, an MSE loss, a recognition loss, and an image similarity loss; constructing a cross-entropy loss function, the cross-entropy loss function being used to adjust parameters of a video classifier, the video classifier being an assembly for adjusting the sharpness of the pre-generated video in the interactable video generation model so that it gradually approaches the actual video; and performing parameter adjustment for each assembly in the interactable video generation model using the reconstruction loss function and the cross-entropy loss function.

[0015] Preferably, the step of adjusting parameters for each assembly in the interactive video generation model using the reconstruction loss function and the cross-entropy loss function includes: The method includes calculating gradients for parameters of each assembly in the interactive video generation model using the reconstruction loss function, and adjusting the parameters of each assembly in the interactive video generation model using a gradient descent method.

[0016] A real-time interactive video generation system according to a second aspect of the present application comprises: a data acquisition module configured to acquire training data including interactive information and video data corresponding to the interactive information; a data pre-processing module configured to pre-process the training data to obtain target training data; a model training module configured to complete training for the interactive video generation model by performing model training based on the target training data, the model training including: a model pre-training process including training a low-resolution image reconstruction model and training a super-resolution model; and a video pre-generation process including performing video pre-generation based on the interactive video generation model, the interactive video generation model including: a model training module having the low-resolution image reconstruction model and the super-resolution model that have been trained; A video generation module including the interactable video generation model whose training has been completed, configured to perform interactive video generation using the interactable video generation model whose training has been completed, wherein the interactable video generation model whose training has been completed includes a video generation module including the training of the resolution image reconstruction model whose training has been completed and the super-resolution model. [Effects of the Invention]

[0017]

[0010] According to the above, a real-time interactive video generation method according to the present application includes the steps of: acquiring training data including interactive information and video data corresponding to the interactive information; pre-processing the training data to acquire target training data; and completing training of the interactive video generation model by performing model training based on the target training data, wherein the model pre-training process includes training a low-resolution image reconstruction model and a super-resolution model; and a video pre-generation process includes performing video pre-generation based on the interactive video generation model, wherein the interactive video generation model includes the low-resolution image reconstruction model and the super-resolution model after training; and generating an interactive video using the interactive video generation model after training. With the above technical configuration, the present application solves the problem that conventional video generation technologies are unable to perform interactive adjustments in real time. Hereinafter, in order to more clearly explain the technical configuration of the present application, drawings that need to be used in the embodiments will be briefly introduced, and those skilled in the art can derive other drawings based on these drawings without any creative effort. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a flowchart of the present real-time interactive video generation method. [Figure 2] FIG. 1 is a data processing schematic diagram for training a low-resolution image reconstruction model and a super-resolution model in the present real-time interactive video generation method. [Figure 3] 1 is a schematic diagram of data processing for training an interactable video generation model in the real-time interactable video generation method of the present application; FIG. [Figure 4]FIG. 1 is a schematic diagram of the data processing for training a Transformer in the present interactable video generation model. [Figure 5] 1 is a schematic diagram of the present real-time interactive video generation system; DETAILED DESCRIPTION OF THE INVENTION

[0019] The following clearly and completely describes the technical configurations of the embodiments of the present invention with reference to the drawings in the embodiments of the present invention, and the described embodiments are only some of the embodiments of the present invention, not all of the embodiments, and all other embodiments that can be obtained by a person skilled in the art based on the embodiments of the present invention without any creative effort are all within the scope of protection of the present invention.

[0020] In a particular interactive scene, the captured continuous interactive operations are used to generate continuous video images, and the content included in the generated video images is matched to the interactive operations. The interactive scenes include, but are not limited to, scenes using keyboard or mouse clicks, gamepad operations, voice control, motion capture using haptic devices, motion capture using video cameras, and brain-computer interfaces (BMIs). The video images and content are generated by the interaction, and changes in the content occur in response to changes in the interaction, including, but not limited to, video content such as game screens and scene guides.

[0021] For example, in a certain game, operations are provided using arrow keys on a keyboard, and the game screen generates corresponding images according to changes in keyboard input. Continuous keyboard inputs and corresponding image content are collected to train an interactive video generation model. Continuous keyboard interactions are input into the interactive video generation model to generate continuous image content (i.e., video content), which is related to a specific scene. That is, a certain interactive video generation model can generate video content for a specific scene. When video content for a new scene needs to be generated, new data needs to be collected to train a new interactive video generation model.

[0022] Currently, conventional techniques for generating video include several types:

[0023] In the first technical solution, the conventional video generation process includes the steps of (1) inputting text information into a Decoder model of a Transformer neural network and encoding the text information to generate encoding features of corresponding frame images; (2) using a Decoder model of a VQVAE neural network to decode the encoding features of the frame images into a current frame image; (3) downsampling the generated current frame image and inputting it into an Encoder model of a VQVAE neural network to obtain encoding information of a generated image; (4) inputting the text information and the encoding information of the generated image into a Decoder model of a Transformer neural network to obtain encoding features of the corresponding next frame image; and (5) repeating steps (2) to (4) until a predetermined number of frames is reached, to output a complete video.

[0024] In the second technical proposal, the process of generating a video is as follows: (1) randomly sampling the temporal noise, and inputting the temporal noise through a mapping network into a 3D generator model with a structure similar to StyleGAN2 and StyleGAN3, which then upsamples the initial input from the temporal and spatial dimensions respectively to generate a low-resolution video; and (2) inputting the low-resolution video generated in step (1) into a video super-resolution model to obtain a high-resolution generated video.

[0025] In the third technical solution, video generation can be performed using text information as a condition, or unconditional video generation can be performed. The video generation method according to the third technical solution uses a diffusion model with a 3D-UNet as its backbone, samples noise from a standard Gaussian distribution, and gradually removes the noise to finally obtain a generated video. During the gradually removing noise process, a pre-trained language model such as BERT or CLIP can be used to extract text features from pre-specified text information, and a self-attention mechanism can be used to inject the text features into the 3D-UNet to provide text condition guidance for video generation. Of course, unconditional video generation can also be performed without using text information.

[0026] As can be seen from the above technical solutions, the first technical solution is conditional video generation using text information, the second technical solution is unconditional video generation without using other information, and the third technical solution may perform conditional video generation using text information or unconditional video generation without using other information. In the above methods, when text information is specified in advance or when the initial state is specified in advance, the generated video content is fixed, i.e., the images of each frame of the generated video are fixed. Each frame image of the generated video is related only to the initial input, and when the input changes during the generation process, i.e., when interactivity occurs, the above methods cannot process it. Furthermore, the output of the second and third technical solutions is in video format, and real-time video stream generation is not possible.

[0027] Based on the above, this embodiment provides the following technical configuration to solve the above problems.

[0028] FIG. 1 is a flowchart of the present real-time interactive video generation method.

[0029] Referring to FIG. 1, this embodiment provides a real-time interactive video generation method, which includes the following steps S100 to S400.

[0030] In step S100, training data including interactive information and video data corresponding to the interactive information is acquired. Specifically, in this embodiment, the training data is continuous interactive information and video data corresponding to the interactive information. Here, the interactive information includes, but is not limited to, keyboard or mouse clicks, gamepad operation, voice control, motion capture using a sensory device, motion capture using a camera, and brain-computer interface. The interactive information can be directly collected and acquired by some external devices, or can be collected by some external devices and extracted from the collected data using related software or algorithms. The interactive video data is video generated through an interactive process, and changes occurring during the interactive process can correspond to changes in the video content. For example, in a racing game, keyboard operations can control the movement of objects on the game screen, and changes in the keyboard operations can cause corresponding changes in the content of the game screen.

[0031] When acquiring training data, interactive operation information and video image data must be recorded simultaneously. The longer the training video, the better, preferably at least 10 minutes, but this is not a specific limitation. The video resolution can be any standard resolution, such as 360P, 480P, 720P, or 1080P, and is not required. The video frame rate can be any commonly used video frame rate, such as 25 frames / second or 30 frames / second, and is not required. Interactive operation information synchronized with the video is simultaneously recorded, and the frame rate of the interactive operation information is equal to or greater than the frame rate of the video and is an integer multiple of the frame rate of the video. For example, if the frame rate of the recorded video is 30 frames / second, the frame rate of the recorded interactive operation information can be 30 frames / second, 60 frames / second, 90 frames / second, 120 frames / second, etc.

[0032] In step S200, the training data is pre-processed to obtain target training data. Specifically, in this embodiment, the training data needs to be pre-processed to facilitate subsequent model training. The step of pre-processing the training data includes: The method includes the steps of performing a downsampling process on the video data to obtain low-resolution images, performing a trimming process on the low-resolution images to obtain a plurality of groups of video segments, and recording the interactive information corresponding to each of the video segments to obtain the target training data, wherein the resolution reduction rate due to the downsampling process is determined by the size of the frame images of the video data, each of the video segments includes T frames, two adjacent video segments are spaced apart by 2 / T frames and have an overlap of 2 / T frames, the frame rate of the interactive information recorded in each of the video segments is equal to or greater than the number of frames corresponding to the video segment, and the frame rate of the interactive information recorded in each of the video segments is an integer multiple of the number of frames corresponding to the video segment.

[0033] For example, to train the image reconstruction model and the super-resolution model S, it is necessary to extract images of all frames from the collected video, and the resolution of these frame images is the same as the resolution of the video, which is denoted here as native resolution x. The native resolution x is downsampled by a predetermined percentage to obtain a low resolution x. L The downsampling rate should be determined according to the size of the frame image. For example, if the video frame image is 360P or 480P, the downsampling rate is 2, that is, the downsampled low resolution x L The length and width of the video frame is 1 / 2 of the native resolution x. If the video frame image is 720P or 1080P, the downsampling rate is 4, that is, the downsampled low resolution x L The length and width of is 1 / 4 of the native resolution x. The higher the resolution of the original image, the higher the downsampling rate.

[0034] In addition, the low resolution x Land native resolution x is the format of the image.

[0035] To train an interactive video generation model, we need to trim the original video into multiple segments. Let us denote the image portion of each video segment as v, where v is the set of frame images [x 1 ,…x t ,…x T ], where T is the number of frames included in each video segment, and t is the index of a frame image in the video segment. When trimming a video into segments, if each segment contains T frames and the interval between adjacent segments is T / 2 frames, the adjacent segments will have an overlap of T / 2 frames. The interactive operation information corresponding to a video segment v is expressed as I=[I 1 ,…I t ,…I T ]. Each frame image x t is the corresponding interactive operation I t When recording data, the frame rate of the interactive operation information is equal to or higher than the video frame rate, and the frame rate of the interactive operation information is an integer multiple (denoted as N) of the video frame rate, so that each frame of the video image represents a group of interactive operations I t =[I t 1,I t 2,…,I t n ,…,I t N ], where n corresponds to x t is the index of the interactive operation information corresponding to the frame image, and I t n is the interactive information vector. Due to the variety of interactive formats, I t ncan be discrete or continuous. For example, if the interactive information is collected by clicking a keyboard or mouse or by operating a gamepad, the interactive information is expressed as a discrete value of whether or not a button is pressed or clicked. In this case, I t n is a set of one-hot encoded vectors, and when interactive information is collected by voice control, haptic motion capture, video motion capture, or brain-computer interface, the interactive information is represented as a set of continuous values. I t n If the number of features in a vector is represented by f, then I t is an f×N matrix. In order to integrate the discrete interactive operation information and the continuous interactive operation information, in this embodiment, I t By performing Gaussian smoothing on the time dimension (i.e., N), the discrete interactive information is converted into continuous interactive information, so that continuous interactive information can be obtained for training and estimation even in different interactive methods.

[0036] In some complex interactive methods, such as voice control, motion capture using sensory devices, motion capture using camera devices, and brain-computer interfaces, the raw data collected by each method may include sound signals, images, electrical signals, etc., and the process of processing the raw data collected by each complex interactive method is not included in the data preprocessing process of this embodiment. The information used in the data preprocessing of the interactive operation information in this embodiment is information generated after these interactive methods themselves process their raw data. For example, when voice control is used as the interactive method, this embodiment does not directly use the sound signal, but instead uses some conventional methods to process the sound signal and uses the processed sound features as the interactive operation information.

[0037] In step S300, model training is performed based on the target training data to complete training for the interactive video generation model. The model training includes a model pre-training process including training a low-resolution image reconstruction model and a super-resolution model, and a video pre-generation process including pre-generating a video based on the interactive video generation model. The interactive video generation model includes the low-resolution image reconstruction model and the super-resolution model that have been trained. Specifically, in this embodiment, step S300 is a model training step, and the model pre-training process is divided into two stages: (1) a pre-training process including training a low-resolution image reconstruction model and training a super-resolution model S, and (2) training an interactive video generation model that generates images frame by frame and can generate a video by merging all the generated images, or output the images to a video stream in real time. The two model training stages in the pre-training process do not need to be in any order. The pre-training process must be performed before the training of the interactive video generation model.

[0038] FIG. 2 is a data processing schematic diagram for training the low-resolution image reconstruction model and the super-resolution model in the present real-time interactive video generation method.

[0039] As can be seen from FIG. 2, training a low-resolution image reconstruction model includes the following technical proposals:

[0040] The low-resolution image reconstruction model is the image encoder E M and image decoder D MIt is composed of neural network basic structures such as a convolution layer, a fully connected layer, an activation layer, a pooling layer, and a normalization layer. Several conventional network structures can be used, such as neural networks with encoder and decoder structures such as AutoEncoder, VAE, and VQVAE, and a self-designed image reconstruction model can also be used with the basic structure. Here, there is no requirement for the specific structure of the neural network.

[0041] The low-resolution image reconstruction model is L as input and the image encoder E M By low resolution x L The hidden layer code z x Obtain the hidden layer code z x Image decoder D M The low-resolution x^ reconstructed by inputting L ("x^" indicates an x ​​hat (^ above x). The same applies below.) is obtained, and the reconstructed image x^ L and the input image x L It is desirable that the image encoder E M The hidden layer code z generated by x is used to represent the input image, which is later used to train and estimate the interactive video generation model.

[0042] To train the low-resolution image reconstruction model, we use the actual low-resolution x L is used as input and training data. Image encoder E M and image decoder D M After passing through, the model is reconstructed into a low-resolution x^ L and outputs the reconstruction loss function L R (x L ,x^ L) is calculated. The reconstruction loss function uses MAE loss (L1 loss), MSE loss (L2 loss), recognition loss, image similarity loss, or any other loss function that can represent image differences. After calculating the reconstruction loss function, a backpropagation process is performed. Based on the calculated loss function, the image encoder E M and image decoder D M The gradient is calculated for the parameters of each assembly in the low-resolution image reconstruction model, and the gradient descent method is used to optimize and update the parameters of each assembly in the low-resolution image reconstruction model. The model pre-training process includes multiple rounds, and each round performs the above model pre-training process and stops when a training termination condition is reached, completing the model training.

[0043] When using different neural networks, the loss functions for the calculations can be different.

[0044] Referring to FIG. 2, training the super-resolution model S includes the following technical scheme:

[0045] The super-resolution model S can process an input low-resolution image to obtain a high-resolution image corresponding to the content of the low-resolution image so that the sharpness of the input low-resolution image is improved. The super-resolution model S is composed of a neural network basic structure such as a convolutional layer, a fully connected layer, an activation layer, a pooling layer, and a normalization layer. A conventional neural network for super-resolution tasks can be used as the super-resolution model S, or a self-designed super-resolution model S can be used as the basic structure. There is no requirement for the specific structure of the neural network. Conventional super-resolution models S include, but are not limited to, HAT, SwinIR, LTE, etc.

[0046] The Super Resolution Model S contains a series of upsampling layers inside, and LIt takes as input, and after upsampling, outputs a reconstructed native resolution x^, which is preferably as close as possible to the actual native resolution x.

[0047] To train the super-resolution Model S, we use the actual low-resolution x L is used as input, and the corresponding native resolution x is used as training data. After passing through the super-resolution model S, the reconstructed native resolution x^ is output, and the reconstruction loss L R Calculate (x, x^). The reconstruction loss function can be MAE loss (L1 loss), MSE loss (L2 loss), recognition loss, image similarity loss, or any other loss function that can represent image differences. Optimize the reconstruction loss function to ensure that the reconstructed image output from the super-resolution model S has the same content as the actual native resolution image.

[0048] Note that the reconstructed native resolution x̂ is the format of the image.

[0049] Super resolution classifier D S By optimizing the super-resolution classifier D, it is possible to determine whether the input image is an actual image or a reconstructed image, and to make the sharpness of the reconstructed image output from the super-resolution model S similar to that of the actual image. S can optimize the super-resolution model S so that the image output from the super-resolution model S resembles the actual image. S It consists of the basic structure of a neural network, including convolutional layers, fully connected layers, activation layers, pooling layers, and normalization layers. It can use the classifier structure of a conventional generative adversarial network (GAN), or it can use a self-designed classifier in the basic structure.

[0050] Super resolution classifier D Sis used to determine whether the input image is an actual native resolution image or a reconstructed native resolution image, and since it is a model for binary classification, the usual cross-entropy loss function can be used. The cross-entropy loss function L S is as shown in equation (1), where x i is the super-resolution classifier D S is the input image, and y i is the tag of the image. S Input image x i If is the actual native resolution image, then y i is 1, and the super-resolution classifier D S If the input image to is a reconstructed native resolution image, then y i is 0. D S (x i ) is the super-resolution classifier D S It is the output after an image is input to , and its value represents the probability that the input image is the actual image. In addition to using the cross-entropy loss function for classification of the classifier, other loss functions for classification may also be used.

number

[0051] The above reconstruction loss L R and the cross entropy loss L of the classifier S After calculating the loss function, the backpropagation process is performed. Based on the calculated loss function, the gradient is calculated for the parameters of each assembly in the model, and the parameters of each assembly in the model are optimized and updated using the gradient descent method. Note that the above reconstruction loss L R is used only to optimize the parameters of the super-resolution model S, and the classifier loss L S are the parameters of the super-resolution model S and the super-resolution classifier D SThe model pre-training process includes multiple rounds, each of which performs the above model pre-training process and stops when the training end condition is reached, completing the model training.

[0052] FIG. 3 is a schematic diagram of data processing for training a real-time interactive video generation model in the real-time interactive video generation method of the present application.

[0053] Referring to FIG. 3, the technical scheme for training an interactive video generation model is as follows.

[0054] Here, the video pre-generation process (i.e., training an interactive video generation model) includes the steps of generating image hidden layer features corresponding to the initial frame of the low-resolution image; inputting the image hidden layer features and interactive information corresponding to the low-resolution image into a transformer of the interactive video generation model to generate target hidden layer features corresponding to the first frame image; performing image reconstruction on the target hidden layer features to obtain a reconstructed low-resolution image; and performing super-resolution processing on the reconstructed low-resolution image to obtain a pre-generated first frame image corresponding to the pre-generated video, thereby generating frame-by-frame images corresponding to the pre-generated video, wherein the resolution of the pre-generated first frame image is the same as the resolution of the video data.

[0055] Transformer Encoder-Decoder M in Figure 3 V and Interactive Information Encoder E I is considered a transformer in the above sense.

[0056] The interactive video generation model includes an interactive information encoder, E I , Transformer encoder-decoder MV , image encoder E M , image decoder D M and assembly with the Super Resolution Model S. Image Encoder E M , image decoder D M The super-resolution model S is obtained through the above pre-training process. Therefore, the process of training an interactive video generation model starts with the interactive information encoder E I and Transformer encoder-decoder M V Only the

[0057] FIG. 4 is a schematic diagram of the data processing for training the Transformer in the interactive video generation model of this application.

[0058] Referring to FIG. 4, furthermore, in some embodiments, the Transformer includes an interactive information encoder, a Transformer encoder, and a Transformer decoder, so that the step of generating target hidden layer features corresponding to the first frame image further includes the steps of inputting the interactive information corresponding to the low-resolution image into the interactive information encoder to obtain interactive hidden features corresponding to the interactive information of the low-resolution image, inputting the image hidden layer features into the Transformer decoder, the Transformer encoder transmitting the interactive hidden features to the Transformer decoder, and the Transformer decoder generating target hidden layer features corresponding to the image frame based on the interactive hidden features and the image hidden layer features.

[0059] However, to avoid confusion, in the written description, the Transformer encoder ME in Figure 4 v is called the confusion encoder, and the Transformer decoder MD v is called a confusion decoder.

[0060] Illustratively, an interactive information encoder E I is the preprocessed interactive operation information I t as input and interactive operation information I t The network feedforward and nonlinear transformation are performed on the interactive information encoder E to obtain the hidden layer features of the interactive operation information. I is composed of basic neural network structures such as convolutional layers, fully connected layers, activation layers, pooling layers, and normalization layers. In Figures 3 and 4, the interactive operation information encoder E I The input is all interactive operation information I t is the interactive operation information corresponding to the t-th frame of the video to be generated. In practice, the interactive information encoder E I The input can be in a variety of forms.

[0061] The above different video tasks can be divided into (1) offline tasks and (2) real-time tasks. In the offline tasks, a set of known interactive operation information I = [I 1 ,…I t ,…I T ] is specified in advance, where T is the number of frames of the video to be generated, and t is the t-th frame of the generated video. The known interactive operation information I is used to directly generate the corresponding video v. In real-time tasks, the interactive information I of the history frame / current frame is expressed as I = [I 1 ,…I t ] is calculated in advance, and the video frame x^ corresponding to the current time t is calculated. t The generated video frames are output in the form of a video stream, enabling real-time video generation.

[0062] In the offline task, the interactive operation information corresponding to all frames is known, so the interactive information encoder E Imay use only the interactive operation information corresponding to the t-th frame, or may use the interactive information corresponding to the t-th frame and some frames before and after it (e.g., [...,I t-2 ,…I t-1 ,I t ,I t+1 ,I t+2 ,...]) may also be used. When using interactive operation information of multiple frames, the interactive operation information of multiple frames may be synthesized (spliced) in the time dimension.

[0063] In real-time tasks, only the interactive operation information corresponding to the past frame and the current frame is known. Therefore, the interactive information encoder E I may use only the interactive operation information corresponding to the t-th frame, or may use the interactive operation information corresponding to the t-th frame and some frames before it (e.g., [...,I t-2 ,…I t-1 ,I t ]) may also be used. When using interactive operation information of multiple frames, the interactive operation information of multiple frames may be synthesized (spliced) in the time dimension.

[0064] Transformer Encoder-Decoder M V The structure is a standard Transformer model, so we will not explain it in detail here. v The Transformer decoder MD encodes the hidden layer features of the input interaction information and uses its internal self-attention mechanism to calculate the importance of each step of the interaction information in the sequence to the generation of the corresponding frame. v is the encoder ME v The output of is used as input, and the hidden layer feature z^ of the image generated by the previous frame is used as input. t-1 ("z^" indicates a z hat (^ on z). The same applies below.) is used as input. The generated previous frame image is sent to the image encoder E MWhen input to the hidden layer feature z^ of the image t-1 can be obtained. Transformer decoder MD v The function of is to combine the interactive operation information and the information of the generated image of the previous frame to generate the hidden layer feature z^ of the current frame. t M and generate the image decoder D M Current frame image by x^ t L Then, the image sharpness is improved by the super-resolution model S, and finally the image of the current frame is generated as the frame x^ t The goal is to obtain

[0065] Furthermore, after the generation of one segment of video is completed, this embodiment further optimizes the technical solution, specifically, the steps of: merging all generated image frames to obtain the pre-generated video; calculating a reconstruction loss function of the pre-generated video and an actual video corresponding to the video data, where the reconstruction loss function includes MAE loss, MSE loss, recognition loss and image similarity loss; and constructing a cross-entropy loss function, where the cross-entropy loss function is used to adjust parameters of a video classifier, and the video classifier determines whether the sharpness of the pre-generated video is greater than that of the actual video in the current interactive video generation model. and performing parameter adjustment for each assembly in the interactive video generation model using the reconstruction loss function and the cross-entropy loss function, wherein the step of adjusting the parameters for each assembly in the interactive video generation model using the reconstruction loss function and the cross-entropy loss function includes calculating gradients for parameters of each assembly in the interactive video generation model using the reconstruction loss function, and adjusting the parameters of each assembly in the interactive video generation model using a gradient descent method.

[0066] Specifically, in this embodiment, the interactive information encoder E of the interactive video generation model I , Transformer encoder-decoder M V To train , the preprocessed interactive operation information is used as input, and the actual video segment v corresponding to the interactive operation is used as training information. After passing through the interactive video generation model, the generated video v^ (where "v^" indicates a ^ over v, and so on) with the same resolution as the actual video segment is output, and the reconstruction loss L between the generated video and the actual video is calculated. R Calculate (v, v^). The reconstruction loss function uses MAE loss (L1 loss), MSE loss (L2 loss), recognition loss, image similarity loss, or any other loss function that can represent image differences. The reconstruction loss function is optimized to ensure that the video generated by the interactable video generation model has the same content as the actual video.

[0067] And the video classifier D V The video classifier D determines whether the input video is real or generated, and the sharpness of the video output by the interactive video generation model is made closer to the real video. The principle and loss function calculation formula are the same as those of the super-resolution classifier DS, so they will not be described again. V The difference between the super-resolution classifier DS and the 2D image is that the super-resolution classifier D S and the 3D video is input to the video classifier D V The input to the video classifier D V is used to determine whether the input video is a real video or a generated video. Since it is a binary classification model, the cross-entropy loss function L V Using the video classifier D V can be optimized.

[0068] The above reconstruction loss L R and the cross entropy loss L of the classifier VAfter calculating L, the error backpropagation process is performed. Based on the calculated loss function, the gradient is calculated for the parameters of each assembly in the interactive video generation model, and the gradient descent method is used to optimize and update the parameters of each assembly in the model. Note that the above reconstruction loss L R Interactive Information Encoder E I and Transformer encoder-decoder M V is used only to optimize the classifier loss function L V Interactive Information Encoder E I , Transformer encoder-decoder M V and video classifier D V It is used to optimize the three parameters. M , image decoder D M The parameters of the super-resolution model S are not updated during this training process. The model pre-training process includes multiple rounds, and each round executes the above model pre-training process and stops when a training end condition is reached, completing the training of the interactive video generation model.

[0069] Referring to Figure 3, the step of performing image reconstruction on the target hidden layer features to obtain a reconstructed low-resolution image specifically involves inputting the target hidden layer features into an image reconstruction decoder, and performing image reconstruction on the target hidden layer features by the image reconstruction decoder to obtain the reconstructed low-resolution image, and the image reconstruction decoder is a decoder of the low-resolution image reconstruction model whose training has been completed.

[0070] Referring to Figure 3, the step of performing super-resolution processing on the reconstructed low-resolution image to obtain a pre-generated first frame image corresponding to the pre-generated video specifically involves inputting the reconstructed low-resolution image into the super-resolution model S whose training has been completed, and performing super-resolution processing on the reconstructed low-resolution image using the super-resolution model S to obtain each frame image corresponding to the pre-generated video whose resolution is the same as the video data.

[0071] In order to indicate that the interactive video generation model is equipped with the low-resolution image reconstruction model after training, the image encoder E in FIG. M and image decoder D M are referred to as the image reconstruction encoder and the image reconstruction decoder.

[0072] Furthermore, after the pre-generated first frame image is generated, the generation of subsequent image frames depends on the data of the immediately preceding frame image. Specifically, when generating the pre-generated image of the t-th frame, the image frame of the t-1-th frame is input to the image reconstruction encoder to obtain the image hidden layer features of the image frame of the t-1-th frame and obtain the interactive information corresponding to the image frame of the t-1-th frame. The image hidden layer features and the interactive information corresponding to the image frame of the t-1-th frame are input to the transformer of the interactive video generation model to obtain the target hidden layer features corresponding to the image frame of the t-1-th frame. The target hidden layer features corresponding to the image frame of the t-1-th frame are input to the image reconstruction decoder. The image reconstruction decoder performs image reconstruction using the target hidden layer features of the image frame of the t-1-th frame to obtain a reconstructed low-resolution image corresponding to the image frame of the t-1-th frame. The reconstructed low-resolution image corresponding to the image frame of the t-1-th frame is input to the super-resolution model S whose training has been completed. The super-resolution model S performs super-resolution processing on the reconstructed low-resolution image of the image frame of the t-1-th frame to obtain the pre-generated image frame of the t-1-th frame.

[0073] Step S300 can be summarized as follows:

[0074] (1) Initial frame hidden layer feature z 0 The generation method is as follows: (I) In the process of pre-training the image reconstruction model, the hidden layer features of all known images are obtained, and the hidden layer feature of one image is arbitrarily selected to generate the hidden layer feature z of the initial frame. 0 (II) After pre-training the image reconstruction model, the prior distribution of the hidden layer features is known, and the hidden layer features z of the initial frame are calculated from the prior distribution of the hidden layer features. 0 There are two types of sampling:

[0075] (2) Initial frame hidden layer feature z 0 and interactive operation information I corresponding to the first frame 1 The hidden layer features of Transformer encoder-decoder M V The hidden layer feature z^ is input to reconstruct the first frame image. 1 M and obtain the hidden layer feature z^ 1 M Image decoder D M and the reconstructed low-resolution image x^ 1 L and obtain this low-resolution image x^ 1 L is input to the super-resolution model S to generate the native resolution image frame x^ 1 Once this is obtained, the generation of the first frame of the video is complete.

[0076] (3) For the t-th frame of the generated video, the video image x^ of the previous frame generated is t-1 Image Encoder E M and input the previous frame image x^ t-1 The hidden layer code z^ t-1 and obtain the hidden layer code z^ t-1 and the interactive operation information I corresponding to the current frame tThe hidden layer features of Transformer encoder-decoder M V and the hidden layer feature z^ to reconstruct the current frame image. t M and obtain the hidden layer feature z^ t M Image decoder D M and the reconstructed low-resolution image x^ t L and obtain this low-resolution image x^ t L is input to the super-resolution model S to generate the native resolution image frame x^ t Once the tth frame of the video is obtained, the generation of the tth frame of the video is completed.

[0077] (4) Repeat step (3) until a termination condition is reached. Different video generation tasks have different termination conditions. For offline tasks, the termination condition is typically when the number of generated video frames reaches a preset value, at which point all generated frames are merged into a video v^. For real-time tasks, the generated video frames are output to a video stream, and the termination condition is typically when the user terminates the generation process. During training, the termination condition is when the number of generated video frames reaches the number of frames in the training video segment.

[0078] Illustratively, this example provides the following model training example:

[0079] (1) A game is played using a keyboard, and a game video is recorded along with keyboard operations. The video is 30 minutes long, has a resolution of 512x512 pixels, and has a frame rate of 30 frames per second. The frame rate of the keyboard operation recording is 300 frames per second, i.e., each frame of the video image corresponds to 10 frames of keyboard operation recording.

[0080] (2) Downsample the video with a native resolution of 512x512 pixels to a resolution of 256x256 pixels, extract all frame images from the original video with a resolution of 512x512 pixels, extract all frame images from the downsampled video with a resolution of 256x256 pixels, and trim the complete 30-minute video into several video segments, each video segment containing 16 frame images, with the trimming interval between two adjacent video segments being 8 frames. At the same time, trim the keyboard interactive recording data, each video segment containing 160 frames of interactive data, with each frame image corresponding to 10 frames of interactive data. Perform Gaussian smoothing with a mean value of 0 and a variance of 3 on the interactive data corresponding to each frame.

[0081] (3) Using the downsampled frame images with a resolution of 256 × 256 pixels as input data and training data, the image reconstruction model VQVAE is trained and the image encoder E M and image decoder D M and obtain the prior distribution of the image hidden layer features.

[0082] (4) The downsampled frame images with a resolution of 256 × 256 pixels are used as input data, and the frame images with a native resolution of 512 × 512 pixels are used as training data to train the SwinIR super-resolution model S.

[0083] (5) Extract hidden layer features from the previous frame generated image using the VQVAE encoder (the hidden layer features of the initial frame are obtained by sampling from the prior distribution of image hidden layer features obtained by pre-training the VQVAE), and input the interactive operation information corresponding to the current frame, which has a length of 10, into the interactive information encoder to obtain the interactive information hidden layer features. Then, combine the generated hidden layer features of the previous frame image and the interactive information hidden layer features into the Transformer encoder-decoder M. Vto generate hidden layer features of the current frame image, and then the hidden layer features of the current frame image are input to the image decoder D M to generate a low-resolution image of the current frame (resolution 256 × 256 pixels), and then input this low-resolution image into the pre-trained SwinIR super-resolution model S to obtain an image with a native resolution of 512 × 512 pixels, which is used as the generated current frame image.

[0084] (6) After repeating the process (5) 16 times, the generated video segments can be obtained. The generated video segments and the actual video segments are used to calculate the associated loss function until the training is completed. The associated loss function is used to calculate the interactive information encoder and the Transformer encoder-decoder M V After optimizing the parameters, we obtain a trained, interactive video generation model.

[0085] In step S400, the interactive video is generated using the interactive video generation model that has been trained. Specifically, in this embodiment, in the process of generating the interactive video, the data processing process of the interactive video generation model is as follows: (i) Interactive information encoder and Transformer encoder-decoder M V The calculation process of is placed in one process, and the input queue of this process is established using the output queue of process (iv). (ii) Image decoder D M The calculation steps are placed in one process, and the input queue of the process (i) is established by using the output queue of the process (i). (iii) The calculation process of the super-resolution model S is arranged in one process, and the input queue of the process is established by using the output queue of process (ii). (iv) Image Encoder E MThe calculation steps of (a) and (b) are arranged in one process, and the input queue of this process is established by using the output queue of process (ii). (v) The final video output process is placed in one process, and its input queue uses the output queue in process (iii) to output each generated frame image to a video file or video stream.

[0086] Parallel execution can improve the real-time performance of the model estimation process.

[0087] Illustratively, this embodiment provides an example of generating an interactive video as follows.

[0088] (1) If a group of interactive operation sequences with a known length is specified in advance and the length is 3000 frames, the number of frames in the corresponding generated video will be 300 frames, and the length of the video will be 10 seconds.

[0089] (2) The hidden layer features of the initial frame are sampled and obtained from the prior distribution of the image hidden layer features obtained by pre-training the VQVAE, and the hidden layer features of the initial frame and the interactive operation information corresponding to the generated first frame image are sent to the Transformer Encoder-Decoder M V The hidden layer features of the first frame image are input into the parameters to obtain hidden layer features of the first frame image, and the hidden layer features of the first frame image are input into the VQVAE decoder to obtain a generated low-resolution first frame image, and the low-resolution image is input into the SwinIR super-resolution model S to obtain a generated native-resolution first frame image, i.e., obtain the first frame image of the generated video.

[0090] (3) From the generation of the second frame image, the low-resolution image of the previous frame generated is first input to the VQVAE encoder to obtain the hidden layer features of the previous frame image, and then, according to the interactive information operation corresponding to the generation of the current frame image, the Transformer encoder-decoder M VThe current frame image is generated in native resolution by passing through the parameters, VQVAE decoder, and SwinIR super resolution model S in sequence.

[0091] (4) Repeat step (2) until a 300-frame video is generated. The generated video, which is an integrated image of the 300 frames, is output and saved.

[0092] FIG. 5 is a schematic diagram of the present real-time interactive video generation system.

[0093] Referring to FIG. 5 , this embodiment further provides a real-time interactive video generation system, which includes a data acquisition module 1, a pre-processing module 2, a model training module 3 and a video generation module 4.

[0094] The data acquisition module 1 is configured to acquire training data including interactive information and video data corresponding to the interactive information. Specifically, in this embodiment, the data acquisition module 1 is for executing all the logic flows for acquiring training data in the above-mentioned real-time interactive video generation method.

[0095] The pre-processing module 2 is configured to pre-process the training data to obtain target training data. Specifically, in this embodiment, the data pre-processing module 2 is for realizing the entire logic flow of pre-processing the training data in the above-mentioned interactive video generation method.

[0096] The model training module 3 is configured to perform model training based on the target training data to complete training for the interactive video generation model, the model training including a model pre-training process including training a low-resolution image reconstruction model and training a super-resolution model, and a video pre-generation process including performing video pre-generation based on the interactive video generation model, and the interactive video generation model includes the low-resolution image reconstruction model and the super-resolution model that have been trained. Specifically, in this embodiment, the model training module 3 is for realizing all the logical flows for performing model training in the above-mentioned interactive video generation method.

[0097] The video generation module 4 includes the interactive video generation model after completion of training, and is configured to perform interactive video generation using the interactive video generation model after completion of training, where the interactive video generation model after completion of training includes the resolution image reconstruction model after completion of training and the super-resolution model. Specifically, in this embodiment, the video generation module 4 is for realizing all the logical flows for generating interactive videos in the above-mentioned interactive video generation method.

[0098] This embodiment has the following advantages. The real-time interactive information is used as a condition for generating low-resolution video frames, and the super-resolution model is used to increase the size of the generated low-resolution frame images to obtain high-resolution, clear video frame images. In this process, the process of generating low-resolution video frame images and the process of increasing the sharpness of the video frame images through super-resolution are performed in parallel, so that the generated video stream can be output in real time. [Prior art documents] [Patent documents]

[0099] [Patent Document 1] Chinese Patent Application Publication No. 117135331

Claims

1. obtaining training data including interactive information and video data corresponding to the interactive information; pre-processing the training data to obtain target training data; a step of completing training for the interactive video generation model by performing model training based on the target training data, the model training including a model pre-training process including training a low-resolution image reconstruction model and training a super-resolution model, and a video pre-generation process including performing video pre-generation based on the interactive video generation model, the interactive video generation model having the low-resolution image reconstruction model and the super-resolution model that have been trained; and generating an interactive video using the trained interactive video generation model. A real-time interactive video generation method comprising:

2. The step of pre-processing the training data includes: performing a downsampling process on the video data to obtain a low resolution image; performing a cropping process on the low-resolution image to obtain a set of video segments; and recording the interactive information corresponding to each of the video segments to obtain the target training data; a rate of resolution reduction due to the downsampling process is determined by a size of a frame image of the video data; Each of the video segments includes T frames, and two adjacent video segments are spaced apart by 2 / T frames, and the two adjacent video segments have an overlap of 2 / T frames; a frame rate of the interactive information recorded in each of the video segments is equal to or greater than the number of frames corresponding to the video segment, and the frame rate of the interactive information recorded in each of the video segments is an integer multiple of the number of frames corresponding to the video segment; 2. The method of claim 1, wherein the video is generated in real time and interactively.

3. The step of recording the interactive information corresponding to each of the video segments includes: performing a Gaussian smoothing process in the time dimension on the corresponding interactive information recorded in the video segment; 3. The method of claim 2, wherein the video is generated in real time and interactively.

4. The steps of the video pre-generation process are: generating image hidden layer features corresponding to an initial frame of the low-resolution image; inputting the image hidden layer features and interactive information corresponding to the low-resolution image into a transformer of the interactive video generation model to generate target hidden layer features corresponding to a first frame image; performing image reconstruction on the target hidden layer features to obtain a reconstructed low-resolution image; performing super-resolution processing on the reconstructed low-resolution image to obtain a pre-generated first frame image corresponding to a pre-generated video, thereby generating each frame image corresponding to the pre-generated video, wherein the resolution of the pre-generated first frame image is the same as the resolution of the video data; 3. The method of claim 2, wherein the video is generated in real time and interactively.

5. generating image hidden layer features corresponding to the initial frame of the low-resolution image includes a first method and a second method; The first method includes a step of acquiring hidden layer features of all images in a process of training the low-resolution image reconstruction model, and arbitrarily selecting one feature from all the hidden layer features as the image hidden layer feature; The second method includes, after completing training of the low-resolution image reconstruction model, sampling from a prior distribution of all hidden layer features to obtain the image hidden layer features.

5. The method of claim 4, wherein the video is generated in real time and interactively.

6. The steps of the video pre-generation process are: inputting the target hidden layer features into an image reconstruction decoder, and performing image reconstruction on the target hidden layer features by the image reconstruction decoder to obtain the reconstructed low-resolution image, wherein the image reconstruction decoder is a decoder of the low-resolution image reconstruction model whose training has been completed; inputting the reconstructed low-resolution image into the super-resolution model that has been trained, and performing super-resolution processing on the reconstructed low-resolution image by the super-resolution model to obtain each frame image corresponding to the pre-generated video having the same resolution as the video data; 5. The method of claim 4, wherein the video is generated in real time and interactively.

7. The steps of the video pre-generation process are: When generating a t-th frame pre-generated image, input the t-1-th frame image frame into an image reconstruction encoder to obtain image hidden layer features of the t-1-th frame image frame; Obtaining interactive information corresponding to the image frame of the t-1th frame, and inputting the image hidden layer features and the interactive information corresponding to the image frame of the t-1th frame into a transformer of the interactive video generation model to obtain target hidden layer features corresponding to the image frame of the t-1th frame; inputting target hidden layer features corresponding to the image frame of the t-1th frame into an image reconstruction decoder, and performing image reconstruction on the target hidden layer features of the image frame of the t-1th frame by the image reconstruction decoder to obtain a reconstructed low-resolution image corresponding to the image frame of the t-1th frame; inputting a reconstructed low-resolution image corresponding to the image frame of the t-1th frame into the super-resolution model after training, and performing super-resolution processing on the reconstructed low-resolution image of the image frame of the t-1th frame by the super-resolution model to obtain a pre-generated image frame of the t-1th frame; 5. The method of claim 4, wherein the video is generated in real time and interactively.

8. the transformer includes an interactive information encoder, a transformer encoder, and a transformer decoder; The steps of the video pre-generation process are: inputting the interactive information corresponding to the low-resolution image into an interactive information encoder to obtain interactive hidden features corresponding to the interactive information of the low-resolution image; inputting the image hidden layer features to the Transformer decoder, transmitting the interactive hidden features to the Transformer decoder by the Transformer encoder, and generating target hidden layer features corresponding to the image frame based on the interactive hidden features and the image hidden layer features.

5. The method of claim 4, wherein the video is generated in real time and interactively.

9. The steps of the video pre-generation process are: merging all generated image frames into said pre-generated video; calculating a reconstruction loss function for the pre-generated video and an actual video corresponding to the video data, wherein the reconstruction loss function comprises an MAE loss, an MSE loss, a recognition loss, and an image similarity loss; constructing a cross-entropy loss function, the cross-entropy loss function being used to adjust parameters of a video classifier, the video classifier being an assembly for adjusting the sharpness of the pre-generated video in the interactable video generation model so that it gradually approaches the actual video; and performing parameter adjustment for each assembly in the interactive video generation model using the reconstruction loss function and the cross-entropy loss function.

5. The method of claim 4, wherein the video is generated in real time and interactively.

10. adjusting parameters for each assembly in the interactive video generation model using the reconstruction loss function and the cross-entropy loss function, calculating gradients for parameters of each assembly in the interactive video generation model using the reconstruction loss function, and adjusting the parameters of each assembly in the interactive video generation model using a gradient descent method; 10. The method of claim 9, wherein the video is generated in real time and interactively.

11. A real-time interactive video generation system applied to the real-time interactive video generation method according to any one of claims 1 to 10, comprising: a data acquisition module (1) configured to acquire training data including interactive information and video data corresponding to said interactive information; a data pre-processing module (2) configured to pre-process the training data to obtain target training data; a model training module (3) configured to complete training for the interactive video generation model by performing model training based on the target training data, the model training including a model pre-training process including training a low-resolution image reconstruction model and training a super-resolution model, and a video pre-generation process including performing video pre-generation based on the interactive video generation model, the interactive video generation model having the low-resolution image reconstruction model and the super-resolution model that have been trained; a video generation module (4) including the interactable video generation model after completion of training, configured to perform interactive video generation using the interactable video generation model after completion of training, wherein the interactable video generation model after completion of training includes the resolution image reconstruction model after completion of training and the super-resolution model; A real-time interactive video generation system.

Images