Video reconstruction method based on video variational auto-encoder

Abstract

The invention discloses a video reconstruction method based on a video variational auto-encoder, and relates to the technical field of video reconstruction. According to the method, a one-dimensional time domain residual module is introduced on the basis of an original image decoder to form a pseudo three-dimensional residual module, and a one-way circulation module is added behind the pseudo three-dimensional residual module to form a video decoding module; inputting a to-be-reconstructed video into an original image encoder of the video variational auto-encoder to obtain encoding features of the to-be-reconstructed video in a low-dimensional hidden space; the encoding features are input into a video decoding module of the video variational auto-encoder, and a one-dimensional time domain residual module in a pseudo three-dimensional residual module is utilized to enhance time domain local feature interaction between continuous video frames in the encoding features; and performing time-domain long-distance feature transmission on the video frame output by the pseudo three-dimensional residual module through a hidden state including historical video frame features to obtain a reconstructed video. The method can improve the visual quality of the reconstructed video.

Description

Technical Field

[0001] The present invention relates to the technical field of video reconstruction, and in particular to a video reconstruction method based on a video variational autoencoder. Background Art

[0002] In recent years, artificial intelligence content generation technology (AIGC) combined with large models has developed rapidly in academia and industry. However, while large models bring performance advantages, they also bring tremendous pressure on computing power and resources, becoming a major challenge for the practical application of AIGC. In the field of image generation, latent diffusion generation models (LDM) use variational autoencoders to map pixel space to latent space, thereby converting the original diffusion generation process that needs to be carried out in high-dimensional space to low-dimensional space, which greatly reduces the amount of computation required for the actual application of the diffusion model. Therefore, latent diffusion generation models represented by Stable Diffusion have become another new paradigm for visual content generation algorithms.

[0003] In the prior art, compared with image generation tasks, video generation tasks require larger networks, more data, and a longer training process. In reality, they often face the dilemma of limited resources and lack of video data. Therefore, video generation models usually use existing image generation models as pre-training models. On the one hand, this facilitates network design. On the other hand, pre-training knowledge on images also provides a basis for the training of video generation models, thereby accelerating training and alleviating the dependence on data volume to a certain extent. In view of the success of latent diffusion generation models in image generation tasks, some research works have also attempted to apply latent diffusion to video generation and directly use existing image latent diffusion models as pre-training models. In order to utilize the latent space pre-trained by image generation, the video generation process also needs to perform spatial mapping through the same encoder. These methods usually use the original image autoencoder to encode or decode the video to achieve video reconstruction.

[0004] However, the independent decoding method of each frame in the prior art ignores the temporal correlation between video frames, resulting in poor temporal continuity of the decoding results, introducing flicker, jitter and distortion, and ultimately affecting the quality of the generated video. Summary of the invention

[0005] Based on this, it is necessary to provide a video reconstruction method based on video variational autoencoder to address the above technical problems. This method can improve the visual degradation phenomenon caused by directly using pre-trained image variational autoencoders in video reconstruction and video generation tasks, alleviate the time domain jitter problem caused by the original single frame decoding, and improve the visual quality of the reconstructed video.

[0006] The present invention adopts the following technical solutions:

[0007] The present invention provides a video reconstruction method based on a video variational autoencoder, comprising:

[0008] A one-dimensional time-domain residual module is introduced on the basis of the original image decoder to form a pseudo three-dimensional residual module, and a one-way loop module is added after the pseudo three-dimensional residual module to form a video decoding module;

[0009] The video variational autoencoder network is constructed by the original image encoder and video decoding module;

[0010] Train the video variational autoencoder network to obtain a video variational autoencoder;

[0011] Input the video to be reconstructed into the original image encoder of the video variational autoencoder to obtain the encoding features of the video to be reconstructed in the low-dimensional latent space;

[0012] The encoded features are input into the video decoding module of the video variational autoencoder, and the one-dimensional time domain residual module in the pseudo three-dimensional residual module is used to strengthen the time domain local feature interaction between consecutive video frames in the encoded features. In the unidirectional loop module, the time domain long-distance feature transfer of the video frames output by the pseudo three-dimensional residual module is performed through the hidden state including the historical video frame features to obtain the reconstructed video.

[0013] Preferably, the video decoding module includes multiple groups of pseudo 3D residual modules and unidirectional loop modules; each video frame of the video to be reconstructed corresponds to a group of pseudo 3D residual modules and unidirectional loop modules.

[0014] Preferably, the pseudo three-dimensional residual module includes a two-dimensional spatial residual module and a one-dimensional temporal residual module; using the one-dimensional temporal residual module in the pseudo three-dimensional residual module to enhance the temporal local feature interaction between consecutive video frames in the coding feature includes:

[0015] For any video frame in the coding feature, the video frame and the frames before and after the video frame are input into the corresponding pseudo 3D residual module, and the visual features of the video frame in the spatial domain are obtained through the 2D spatial domain residual module;

[0016] The visual features in the spatial domain are input into the one-dimensional temporal residual module to obtain the temporal local interaction features after the video frame interacts with the adjacent frames in the temporal domain.

[0017] Preferably, the one-dimensional time domain residual module includes a plurality of time domain operation layers with skip connection branches; the visual features in the spatial domain are input into the one-dimensional time domain residual module to obtain the time domain local interaction features after the video frame interacts with the adjacent frames, including:

[0018] The spatial visual features are sequentially input into multiple temporal operation layers, and the temporal local correlation features are obtained by strengthening the temporal local feature correlation between consecutive video frames.

[0019] The visual features in the spatial domain are fused with the local correlation features in the temporal domain through skip connection branches to obtain the local interaction features in the temporal domain.

[0020] Preferably, the channel flow of the time domain operation layer includes: group normalization, Swish activation, random inactivation layer and one-dimensional convolution in the time domain direction.

[0021] Preferably, the input end of each unidirectional cycle module includes two parts, one part is connected to the output end of the corresponding pseudo three-dimensional residual module in the model depth direction, and the other part is connected to the hidden state output end of the unidirectional cycle module corresponding to the previous video frame in the time domain direction; the unidirectional cycle module includes a window mutual attention feature alignment unit, a convolution unit, a channel splitting unit and a jump connection unit. In the unidirectional cycle module, the video frame output by the pseudo three-dimensional residual module is subjected to time domain long-distance feature transfer through the hidden state including the historical video frame feature to obtain a reconstructed video, including:

[0022] For any video frame, the temporal local interaction features of the video frame and the hidden state output by the unidirectional loop module corresponding to the previous video frame are input into the window mutual attention feature alignment unit to obtain the alignment features;

[0023] The video temporal local interaction features, hidden state and alignment features are spliced ​​in three channels to obtain spliced ​​features, and the spliced ​​features are fused and transformed through convolution units to obtain fused features;

[0024] The fusion feature is divided along the feature channel dimension through the channel segmentation unit to obtain the hidden state residual part and the current frame feature residual part;

[0025] The hidden state residual part is added to the hidden state output by the unidirectional cycle module corresponding to the previous video frame through the jump connection unit to obtain the hidden state output by the unidirectional cycle unit;

[0026] The feature residual of the current frame and the temporal local interaction feature of the video frame are added through the jump connection unit to obtain the decoding feature output by the unidirectional cycle unit;

[0027] The reconstructed video is obtained based on the decoded features output by each unidirectional cycle unit.

[0028] Preferably, the window mutual attention feature alignment unit is established based on the shifted window mutual attention mechanism, and the window mutual attention feature alignment unit is formed by stacking two layers of window mutual attention modules; the temporal local interaction features of the video frame and the hidden state output by the unidirectional loop module corresponding to the previous video frame are input into the window mutual attention feature alignment unit to obtain the alignment features, including:

[0029] The temporal local interaction features of the video frame and the hidden state output by the unidirectional loop module corresponding to the previous video frame are group-normalized respectively, and the normalized temporal local interaction features and hidden state are window-partitioned;

[0030] The temporal local interaction features and hidden states after window division are input into the first-layer window mutual attention module. Under each divided window, the temporal local interaction features of the video frame are used as query features, and the hidden state output by the unidirectional loop module corresponding to the previous video frame is used as the key and value of mutual attention. The multi-head mutual attention calculation is performed through the introduced sinusoidal position encoding mechanism. The result of the multi-head mutual attention calculation is fused with the temporal local interaction features of the video frame to obtain the alignment features of the first stage.

[0031] The windows of the aligned features and hidden states in the first stage are shifted, and the aligned features and hidden states after the window shift are input into the second-layer window mutual attention module. The second-layer window mutual attention module performs multi-head mutual attention calculation on the aligned features and hidden states after the window shift, and the results of the multi-head mutual attention calculation are fused with the aligned features in the first stage to obtain the aligned features output by the window mutual attention feature alignment unit.

[0032] Preferably, each group of pseudo three-dimensional residual modules and unidirectional loop modules are connected via a dual-branch upsampling module, which is obtained by expanding the interpolation upsampling module after the spatial domain residual module of the original image decoder; the dual-branch upsampling module includes two branches, one branch includes the nearest neighbor difference and 2D convolution, and the other branch includes 2D convolution and pixel rearrangement, and the method further includes:

[0033] For any video frame, the time-domain local interaction features output by the pseudo 3D residual module are sequentially subjected to the nearest neighbor difference and 2D convolution to obtain the first up-sampled features;

[0034] The time-domain local interaction features output by the pseudo 3D residual module are sequentially subjected to 2D convolution and pixel rearrangement to obtain the second up-sampled features;

[0035] The first up-sampled feature and the second up-sampled feature are fused, and the fused feature is input into the unidirectional loop module.

[0036] Preferably, the video variational autoencoder comprises a plurality of video decoding modules; the plurality of video decoding modules are arranged in series, and the input of the next video decoding module is the output of the previous video decoding module.

[0037] Preferably, training the video variational autoencoder network to obtain a video variational autoencoder includes:

[0038] A consistent initialization strategy is used to initialize the parameters of the video variational autoencoder network while maintaining the spatial domain pre-training knowledge to form a video reconstruction model.

[0039] The video reconstruction model is fine-tuned in the time domain layer on the training video data to obtain a video variational autoencoder.

[0040] The present invention provides a video reconstruction device based on a video variational autoencoder, comprising:

[0041] A construction module is used to introduce a time domain residual module on the basis of the original image decoder to form a pseudo three-dimensional residual module, and a one-way loop module is added after the pseudo three-dimensional residual module to form a video decoding module; a video variational autoencoder network is formed through the original image encoder and the video decoding module; the video variational autoencoder network is trained to obtain a video variational autoencoder

[0042] The encoding module is used to input the video to be reconstructed into the original image encoder of the video variational autoencoder to obtain the encoding features of the video to be reconstructed in the low-dimensional latent space;

[0043] The decoding module is used to input the encoded features into the video decoding module of the video variational autoencoder, use the time domain residual module in the pseudo three-dimensional residual module to strengthen the time domain local feature interaction between consecutive video frames in the encoded features, and in the one-way loop module, transfer the time domain long-distance features of the video frames output by the pseudo three-dimensional residual module through the hidden state including the historical video frame features to obtain the reconstructed video.

[0044] The present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the video reconstruction method based on the video variational autoencoder is implemented.

[0045] The present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned video reconstruction method based on video variational autoencoder when executing the program.

[0046] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0047] In the present invention, firstly, a time domain residual module is introduced on the basis of the original image decoder to form a pseudo three-dimensional residual module, and a unidirectional loop module is added after the pseudo three-dimensional residual module to form a video decoding module. In this way, when decoding the coding features of the video to be reconstructed, the video variational autoencoder can simultaneously consider the time change information between adjacent frames, thereby realizing the interaction of short-distance features in the time domain between adjacent video frames; and, when using the hidden state to transfer the long-distance features in the time domain to each coding feature, the hidden state includes the features of the historical video frames. In this way, by transferring the features of the historical video frames to the current video frame, the continuous process of the video can be better described, the continuity of the reconstructed video is improved, and the time domain jitter problem caused by the original single frame decoding is effectively improved, thereby improving the visual quality of the reconstructed video. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0049] Figure 1 A schematic flow chart of a video reconstruction method based on a video variational autoencoder provided by the present invention;

[0050] Figure 2 A schematic diagram of the structure of a video variational autoencoder provided by the present invention;

[0051] Figure 3 A schematic diagram of the structure of a video decoding unit provided by the present invention;

[0052] Figure 4 A schematic diagram of the structure of a two-dimensional spatial domain residual module and a one-dimensional temporal domain residual module provided by the present invention;

[0053] Figure 5 A schematic diagram of the structure of a one-way circulation module provided by the present invention;

[0054] Figure 6 A structural schematic diagram of a window mutual attention feature alignment unit provided by the present invention;

[0055] Figure 7 A schematic diagram of the structure of a dual-branch upsampling module and a dual-branch sampling module at a video decoder provided by the present invention;

[0056] Figure 8 A schematic diagram of some visualization results of a video variational autoencoder Rec3D-VAE provided by the present invention and four other comparison models on a test set;

[0057] Fig. 9A schematic diagram of some visualization results of another video variational autoencoder Rec3D-VAE provided by the present invention and four other comparison models;

[0058] Fig.10 A schematic diagram of some visualization results of another video variational autoencoder Rec3D-VAE provided by the present invention and four other comparison models on a test set;

[0059] Fig.11 A schematic diagram of the application effect of a video variational autoencoder Rec3D-VAE provided by the present invention in the Hotshot-XL latent diffusion video generation model;

[0060] Fig.12 A schematic diagram of a video reconstruction device based on a video variational autoencoder provided by the present invention;

[0061] Fig.13 A schematic diagram of a computer device for implementing a video reconstruction method based on a video variational autoencoder provided by the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0063] Currently, researchers have noticed the visual degradation caused by applying image variational autoencoders to videos, and have tried to improve the visual quality of generated videos by proposing an additional post-processing model for video Deflicking (VD). However, the neural network filtering method (NF) they proposed requires online training for each video to be reconstructed, and the addition of additional models also makes the video generation process cumbersome. There are also some methods that try to introduce a time domain operation layer (such as adding a convolutional layer or attention layer in the time domain) into the original image decoder model and fine-tune the video data to construct a video decoder. Although this simple method of introducing a model layer is simple, the effect of enhancing video quality still needs to be improved.

[0064] Based on this, the present invention provides a video reconstruction method based on video variational autoencoder, which can improve the visual degradation phenomenon caused by directly using pre-trained image variational autoencoders in video reconstruction and video generation tasks, alleviate the time domain jitter problem caused by the original single frame decoding, and improve the visual quality of the decoded video.

[0065] The technical solutions provided by various embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0066] Figure 1 The present invention is a flow chart of a video reconstruction method based on a video variational autoencoder, which specifically includes the following steps:

[0067] S101, introducing a time domain residual module on the basis of the original image decoder to form a pseudo three-dimensional residual module, adding a unidirectional loop module after the pseudo three-dimensional residual module to form a video decoding module; forming a video variational autoencoder network through the original image encoder and the video decoding module; training the video variational autoencoder network to obtain a video variational autoencoder.

[0068] Among them, the video decoding module includes multiple groups of pseudo three-dimensional residual modules and one-way loop modules; each video frame of the video to be reconstructed corresponds to a group of pseudo three-dimensional residual modules and one-way loop modules. Each group of pseudo three-dimensional residual modules and one-way loop modules can be regarded as a video decoding unit, and the video variational autoencoder includes multiple video decoding modules; multiple video decoding modules are arranged in series, and the input of the next video decoding module is the output of the previous video decoding module; specifically, the input of the pseudo three-dimensional residual module in the first video decoding module is the output of the original image encoder, and the input of the pseudo three-dimensional residual modules in the subsequent video decoding modules are all the output of the one-way loop modules in the previous video decoding module, and the one-way loop module of the last video decoding module outputs the reconstructed video. As Figure 2 As shown, Figure 2 It is a structural diagram of a video variational autoencoder; multiple video decoding modules constitute a video decoder of the video variational autoencoder.

[0069] Specifically, each group of pseudo 3D residual modules and unidirectional loop modules can be regarded as a video decoding unit. Each video decoding unit also includes a dual-branch upsampling module. The input end of each pseudo 3D residual module is connected to the input end of the dual-branch upsampling module. The input end of each unidirectional loop module is respectively connected to the output end of the dual-branch upsampling module of the decoding unit to which it belongs, and the hidden state output end of the unidirectional loop module of the previous video frame. The stacking of multiple unidirectional loop modules in the time domain direction constitutes a unidirectional loop structure to realize long-distance feature transmission in the video time domain, such as Figure 3 As shown, Figure 3 The figure is a structural diagram of a video decoding unit.

[0070] The pseudo three-dimensional residual module includes an original two-dimensional spatial residual module and a one-dimensional time domain residual module designed by the present invention, the input end of the two-dimensional spatial residual module is connected to the output end of the original image encoder, the output end of the two-dimensional spatial residual module is connected to the input end of the one-dimensional time domain residual module, and the output end of the one-dimensional time domain residual module is connected to the feature input end of the one-way loop module; Figure 4 As shown, Figure 4 It is a structural diagram of a two-dimensional spatial domain residual module and a one-dimensional temporal domain residual module; Figure 4 Figure (a) is the structure diagram of the two-dimensional spatial residual module. Figure 4 Figure (b) shows the structure of the one-dimensional time-domain residual module. The two-dimensional spatial residual module is composed of a stack of spatial operation layers with skip-connected branches. The channel flow of each spatial operation layer includes group normalization, Swish activation, and spatial 2D convolution. The one-dimensional time-domain residual module calculates local correlation features of continuous video frames in the time domain. It is composed of four stacked time-domain operation layers with skip-connected branches. The channel flow of each time-domain operation layer includes group normalization, Swish activation, random dropout layer, and one-dimensional convolution in the time domain.

[0071] The unidirectional recurrent structure uses hidden states to perform long-distance feature transfer in the video time domain and is composed of multiple stacked unidirectional recurrent modules; the unidirectional recurrent module includes a window mutual attention feature alignment unit, a convolution unit, a channel splitting unit, and a jump connection unit.

[0072] Take the i-th decoding stage of the decoder (or the i-th video decoding module) as an example. Figure 5 As shown, represents the feature of the t-1th frame output by the pseudo 3D residual module at the i-th decoding stage, F t i represents the features of the t-th frame output by the pseudo 3D residual module at the i-th decoding stage, represents the features of the t+1th frame output by the pseudo 3D residual module at the i-th decoding stage, in F t i In the corresponding one-way loop module, according to F t i And the hidden features output by the one-way loop module corresponding to the t-1th frame Output F t i+1 as well as Specifically, the implementation process of each one-way loop module includes: when calculating the features of the t-th frame of the video, the one-way loop module not only obtains the current feature F t i , and also obtain the hidden state of the previous one-way loop module operation output As input; after the operation of this stage, the feature F of the current time position is output t i+1 As the input of stage i+1, it outputs As the hidden state input of the next unidirectional loop module in the same stage; thus, each calculated video frame feature is accumulated into the hidden state and transmitted in the time dimension as the calculation deepens.

[0073] Inside the unidirectional recurrent module, the window mutual attention feature alignment unit based on the shifted window mutual attention mechanism (SWCA) in SwinTransformer converts the hidden state Align to the current frame feature F t i , get the aligned hidden state Then the current frame state F t i , aligned hidden states and the original hidden state The concatenation is performed along the channel dimension, and the features are fused and transformed through the convolution unit. When output from the convolution unit, the number of channels is transformed to twice the number of channels of the original single frame input. Then, the channel segmentation unit divides the feature map along the feature channel dimension to obtain the hidden state residual part and the current frame feature residual part. The two residual parts are added to the corresponding skip connection features to obtain the feature output F at this stage and time position. t i+1 and hidden state

[0074] The window mutual attention feature alignment unit is composed of two layers of window mutual attention stacked, such as Figure 6 As shown, in the first layer of window mutual attention, the window is divided and then the multi-head mutual attention calculation is performed. In the second layer of window mutual attention, the window shift operation is performed and then the multi-head mutual attention calculation is performed. In addition, in order to alleviate the problem of missing original two-dimensional position information during visual attention calculation, the present invention additionally introduces sinusoidal position encoding (PE) during the original window mutual attention calculation to provide the original image two-dimensional information; specifically, the current frame feature F is calculated during the calculation. t i As the query feature (Query, represented by Q in the figure), the historical hidden state As the key (Key, represented by K in the figure) and value (Value, represented by V in the figure) of mutual attention; additionally introduces sinusoidal position encoding in the original window mutual attention operation to provide the original two-dimensional information of the image; according to the regional similarity between the hidden state and the current frame features, the features of the corresponding position of the hidden state are "moved" and "weighted", aligned to the current features, and the aligned hidden state features are output

[0075] It should be noted that the decoding module itself contains a sufficient number of computational layers and does not use the multi-layer linear perceptron in the original SwinTransformer.

[0076] The interpolation upsampling module after the original two-dimensional spatial residual module is expanded to form a dual-branch upsampling module, such as Figure 7 As shown in the figure, the dual-branch upsampling module adds a parallel branch combined with super-resolution pixel rearrangement on the basis of the original nearest neighbor interpolation to form a dual branch, enriching the features of decoding upsampling and optimizing the decoding quality. The dual-branch upsampling module consists of two parallel branches, one branch first undergoes nearest neighbor interpolation and then 2D convolution, and the other branch first undergoes 2D convolution and then super-resolution pixel rearrangement, and finally the results of the two branches are fused.

[0077] It should be noted that Figure 7 The figure shows the connection between multiple video decoding modules, and the video decoding module includes a pseudo 3D residual module and a dual-branch upsampling module, but does not illustrate the unidirectional loop module. In actual applications, the dual-branch upsampling module in the previous video decoding module and the pseudo 3D residual module in the next video decoding module are connected through the unidirectional loop module in the previous video decoding module.

[0078] Based on the above-mentioned video variational autoencoder network structure, the video variational autoencoder network is trained to obtain a video variational autoencoder. In an exemplary embodiment, the video variational autoencoder network is trained to obtain a video variational autoencoder, including: using a consistent initialization strategy to initialize the parameters of the video variational autoencoder network while maintaining spatial domain pre-training knowledge to form a video reconstruction model; performing time domain layer fine-tuning training on the video reconstruction model on the training video data to obtain a video variational autoencoder.

[0079] In order to load and maintain the spatial domain pre-training knowledge in the video decoder, a special initialization method is required: when the video reconstruction model is initialized, the identity initialization method is used to ensure that the output of the video model is exactly the same as the result of the single-frame operation of the original image model. Specifically, after the original image decoder parameters are fully loaded in the two-dimensional module, the one-way recurrent module in the video reconstruction model and the one-dimensional time domain residual module in the pseudo-three-dimensional module are initialized. Due to the design of the jump connection, only the last layer of fusion convolution in the one-way recurrent module, the last layer of convolution in the one-dimensional residual module, and the convolution parameters of the pixel rearrangement branch need to be initialized to zero, and the remaining parameters are all randomly initialized. In the selection of the pre-trained image reconstruction model, the variational autoencoder inside the high-quality new version of the Stable Diffusion series of the Vincent Image Large Model Stable Diffusion XL (SDXL) can be selected as the pre-trained image variational autoencoder.

[0080] In order to train and test the video reconstruction model, this embodiment constructs a large-scale, high-quality video dataset in an open domain. The specific process is as follows: first, the material website Pexels is selected as the original data source, and the videos on the site are downloaded in batches. The videos on this material website have the characteristics of high resolution and high quality; then, the video automatic reading tool (FFmpeg) is used to read these original videos one by one and filter out videos that cannot be read normally; finally, the long videos are cut to balance the data.

[0081] The video dataset constructed in this embodiment includes 345K video data, of which 1K are used to constitute a test dataset, and the remaining 344K videos constitute a training dataset; the dataset is high-quality, high-volume, and open-domain oriented.

[0082] The video reconstruction model is fine-tuned on the time domain layer on the training video data, specifically including: in terms of training strategy, in order to maintain the original pre-trained image model knowledge, this embodiment only fine-tunes the newly added time domain operation layer, and fixes all the two-dimensional pre-training layers; in addition, since the original latent space needs to be kept unchanged, the encoding module will not participate in the training.

[0083] The loss function of the video reconstruction model fine-tuning training in this embodiment consists of two parts: reconstruction loss and perceptual loss based on VGG16. Since the encoder does not need to be trained, the KL divergence loss term in the general variational autoencoder training is not required in the video reconstruction model training of this embodiment, but the encoding output of the video reconstruction model still needs to be re-parameterized and sampled.

[0084] After the video variational autoencoder is constructed, the video to be reconstructed is input into the encoder of the video variational autoencoder to obtain the encoding features.

[0085] S102, inputting the video to be reconstructed into the original image encoder of the video variational autoencoder to obtain the encoding features of the video to be reconstructed in the low-dimensional latent space.

[0086] Specifically, see Figure 2 The image encoder includes multiple encoding modules, each encoding module corresponds to a video frame, and the video to be reconstructed is input into the original image encoder of the video variational autoencoder. Each encoding module encodes each video frame in the video to be reconstructed respectively to obtain the encoding features of the video to be reconstructed in the low-dimensional latent space.

[0087] S103, input the encoded features into the video decoding module of the video variational autoencoder, use the one-dimensional time domain residual module in the pseudo three-dimensional residual module to strengthen the time domain local feature interaction between consecutive video frames in the encoded features, and in the one-way loop module, transfer the time domain long-distance features of the video frames output by the pseudo three-dimensional residual module through the hidden state including the historical video frame features to obtain the reconstructed video.

[0088] Please refer to Figure 2 When the original image encoder inputs the coding features into the video decoding module, the pseudo 3D residual module of each group of the video decoding module receives the coding features of the corresponding video frame and the adjacent frames of the video frame respectively; for example, if the video frame is the first frame, the adjacent video frame of the video frame can be the second frame, if the video frame is the last frame, the adjacent frame of the video frame is the previous frame of the last frame; if the video frame is an intermediate frame, the adjacent frames of the video frame are the previous frame and the next frame of the video frame.

[0089] See also Figure 4 , using the one-dimensional time domain residual module in the pseudo three-dimensional residual module to strengthen the time domain local feature interaction between consecutive video frames in the coding features, including: for any video frame in the coding features, the video frame and the previous and next frames of the video frame are input into the corresponding pseudo three-dimensional residual module, and the visual features of the video frame in the spatial domain are obtained through the two-dimensional spatial domain residual module; the visual features in the spatial domain are input into the one-dimensional time domain residual module to obtain the time domain local interaction features after the video frame interacts with the adjacent frames in the time domain.

[0090] The video frame and the preceding and following frames of the video frame are input into the corresponding pseudo three-dimensional residual module, and the visual features of the video frame in the spatial domain are obtained through the two-dimensional spatial residual module, including: the video frame and the preceding and following frames of the video frame are input into multiple spatial operation layers in sequence, and feature fusion is performed through the residual structure to obtain the visual features of the video frame in the spatial domain; the visual features in the spatial domain are input into the one-dimensional time domain residual module to obtain the time domain local interaction features after the video frame interacts with the adjacent frames, including: the visual features in the spatial domain are input into multiple time domain operation layers in sequence, and the time domain local correlation features are obtained by strengthening the time domain local feature association between consecutive video frames; the visual features in the spatial domain are fused with the time domain local correlation features through skip connection branches to obtain the time domain local interaction features.

[0091] Optionally, the input end of each unidirectional loop module includes two parts, one part is connected to the output end of the corresponding pseudo 3D residual module in the depth direction of the model, and the other part is connected to the hidden state output end of the unidirectional loop module corresponding to the previous video frame in the time domain direction; please continue to refer to Figure 5 In the unidirectional cycle module, the video frame output by the pseudo three-dimensional residual module is subjected to time domain long-distance feature transfer through the hidden state including the historical video frame feature to obtain a reconstructed video, including: for any video frame, the time domain local interaction feature of the video frame and the hidden state output by the unidirectional cycle module corresponding to the previous video frame are input into the window mutual attention feature alignment unit to obtain an alignment feature; the video time domain local interaction feature, the hidden state and the alignment feature are spliced ​​in three channels to obtain a spliced ​​feature, and the spliced ​​feature is fused and transformed through a convolution unit to obtain a fused feature; the fused feature is feature-divided along the feature channel dimension through a channel segmentation unit to obtain a hidden state residual part and a current frame feature residual part; the hidden state residual part is feature-added with the hidden state output by the unidirectional cycle module corresponding to the previous video frame through a jump connection unit to obtain a hidden state output by the unidirectional cycle unit; the current frame feature residual part and the time domain local interaction feature of the video frame are feature-added through a jump connection unit to obtain a decoding feature of the video frame output by the unidirectional cycle unit; and the reconstructed video is obtained according to the decoding feature output by each unidirectional cycle unit.

[0092] Specifically, the current frame features that have undergone time-domain local feature interaction and the historical hidden state output by the unidirectional loop module of the previous video frame are input into the window mutual attention feature alignment unit to obtain the aligned features; the current frame features, hidden state and aligned features are spliced ​​along the channel dimension, and the spliced ​​features are fused and transformed through the convolution unit so that the current frame features and the historical state features fully interact to obtain the fused features; the fused features are feature-divided along the feature channel dimension through the channel splitting unit to obtain the hidden state residual part and the current frame feature residual part; the hidden state residual part is feature-added with the input historical hidden state through the jump connection unit to obtain the hidden state output at this stage and this time position; the current frame feature residual part is feature-added with the current frame feature through the jump connection unit to obtain the encoding features of the current video frame that fully refer to the historical frame information.

[0093] For details, please refer to Figure 5 As shown, the local interaction feature of the video in time domain is F t i , the hidden state output by the one-way loop module of the previous video frame is The alignment feature is The hidden state output at this stage and time position is The decoding characteristics of the current video frame that fully refers to the historical frame information are

[0094] Optionally, the window mutual attention feature alignment unit is established based on the shifted window mutual attention mechanism. The window mutual attention feature alignment unit is composed of two layers of window mutual attention modules stacked together; please continue to refer to Figure 6, the temporal local interaction features of the video frame and the hidden state output by the one-way cycle module corresponding to the previous video frame are input into the window mutual attention feature alignment unit to obtain the alignment features, including: group normalizing the temporal local interaction features of the video frame and the hidden state output by the one-way cycle module corresponding to the previous video frame, and dividing the normalized temporal local interaction features and hidden state into windows; inputting the temporal local interaction features and hidden state after dividing the windows into the first layer of the window mutual attention module, and in each divided window, using the temporal local interaction features of the video frame as the query feature, and using the hidden state output by the one-way cycle module corresponding to the previous video frame as the query feature. The key and value of mutual attention are calculated, and the multi-head mutual attention is calculated through the introduced sinusoidal position encoding mechanism; the result of the multi-head mutual attention calculation is fused with the time-domain local interaction features of the video frame to obtain the alignment features of the first stage; the aligned features and hidden state windows of the first stage are shifted, and the aligned features and hidden states after the window shift are input into the second-layer window mutual attention module, and the multi-head mutual attention calculation is performed on the aligned features and hidden states after the window shift through the second-layer window mutual attention module, and the result of the multi-head mutual attention calculation is fused with the alignment features of the first stage to obtain the alignment features output by the window mutual attention feature alignment unit.

[0095] In an exemplary embodiment, see Figure 3 and Figure 7 As shown, each group of pseudo three-dimensional residual modules and unidirectional loop modules are connected through a dual-branch upsampling module, and the dual-branch upsampling module is obtained by expanding the interpolation upsampling module after the spatial residual module of the original image decoder; the dual-branch upsampling module includes two branches, one branch includes the nearest neighbor difference and 2D convolution, and the other branch includes 2D convolution and pixel rearrangement. This embodiment includes: for any video frame, the time domain local interaction features output by the pseudo three-dimensional residual module are sequentially subjected to the nearest neighbor difference and 2D convolution to obtain a first upsampling feature; the time domain local interaction features output by the pseudo three-dimensional residual module are sequentially subjected to 2D convolution and pixel rearrangement to obtain a second upsampling feature; the first upsampling feature and the second upsampling feature are fused, and the fused feature is input into the unidirectional loop module.

[0096] That is, the video time-domain interaction features input into the one-way loop module are actually the fusion features output by the dual-branch upsampling module, which will not be described in detail in this embodiment.

[0097] The above describes the construction process and application process of the video variational autoencoder. However, in actual applications, after the video variational autoencoder is constructed, the performance of the video variational autoencoder can be evaluated first. After the video variational autoencoder meets the performance requirements, the video can be reconstructed through the video variational autoencoder.

[0098] It should be noted that the video decoding module in the video variational autoencoder in the present invention is expanded on the basis of the original image decoder, and the video variational autoencoder should also include other parts of the original image decoder, but the embodiments of the present application will not be repeated here. The embodiments of the present invention only describe the decoding process involved in the present invention. After a detailed description of the time domain residual module and the unidirectional loop module, those skilled in the art should be able to understand the meaning of the expansion based on the original image decoder mentioned in the present invention and the necessary steps that the video decoder of the present invention should include.

[0099] In an exemplary embodiment, the finally trained video variational autoencoder (Recurrent 3D Video VAE, Rec3D-VAE) is evaluated using a test set, using evaluation indicators: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM), and compared with other methods, as shown in Table 1. Table 1 shows the video reconstruction results of different variational autoencoders on the test set.

[0100] Table 1

[0101] SD-V2.1-VAE SDXL-VAE SVD-VAE VidRD-VAE Rec3D-VAE PSNR 32.19 32.73 31.35 32.24 33.22 SSIM 0.8717 0.8846 0.8756 0.8728 0.8933

[0102] Please refer to the test results in Table 1. Thanks to the method of combining the unidirectional loop structure with three-dimensional operations, and strengthening the global and local information interaction in the video time domain, the method Rec3D-VAE of the present invention surpasses the other four comparison models in PSNR and SSIM, and the video reconstruction results reach 33.22dB in PSNR and 0.8933 in SSIM.

[0103] Next, in this embodiment, visualization is also used to prove the effectiveness and superiority of the video variational autoencoder provided by the present invention in the task of decoding and reconstructing videos. Figure 8-10As shown, some visualization results of the video variational autoencoder Rec3D-VAE constructed by the present invention and the other four comparison models on the test set are shown. Among them, Input is marked with the original input, SD-V2.1-VAE (Stable Diffusion Version 2.1-Variational Autoencode), SDXL-VAE (Stable Diffusion XL-Variational Autoencode), SVD-VAE (Stable Video Diffusion-Variational Autoencode) and VidRD-VAE (Video Reuse and Diffuse-Variational Autoencode) are respectively the results of the four methods compared in the above Table 1, and Rec3D-VAE is marked with the result of the video reconstruction method of the video variational autoencoder provided by the present invention. Arrows are also used in some results to mark the areas that need to be compared. From the visualization results, the video variational autoencoder Rec3D-VAE provided by the present invention has obvious advantages in the reconstruction effect of details. Figure 8 The advantages of the method of the present invention (represented by Rec3D-VAE in the figure) in restoring clear and detailed information of images are demonstrated; Fig. 9 The advantages of the method of the present invention (represented by Rec3D-VAE in the figure) in alleviating the distortion of the reconstruction results are demonstrated; in addition, unlike the distortion phenomenon that often exists in the results of other methods, the results of the method of the present invention are more natural and beautiful in overall vision ( Fig.10 ), which is due to the fact that the method of the present invention simultaneously strengthens the long-distance information transmission and local feature interaction during video decoding.

[0104] The video variational autoencoder proposed in the present invention is ultimately aimed at the video generation task based on the latent diffusion generative large model. In another embodiment of the present invention, the existing video generation model is combined to explain the role of the video variational autoencoder Rec3D-VAE proposed in the present invention in improving the quality of generated videos. In this example, the open source video generation model Hotshot-XL based on SDXL is used as the generation model in the latent space, and the video variational autoencoder proposed in the present invention is used to replace the autoencoder inside Hotshot-XL to decode and map the generated data in the latent space to the pixel space. Please refer to Fig.11 As shown in the figure, for the same generated data in the latent space, compared with the video results directly generated by Hotshot-XL using SDXL-VAE (see Appendix Fig.11 The first row, marked with a red box), after being decoded by the Rec3D-VAE proposed in this invention (see Appendix Fig.11The second row (marked by the yellow box) has richer texture and more complete shape contours, and the visual quality of the generated video has been improved to a certain extent. This verifies the effectiveness of the improved method of the present invention in video generation tasks, and its plug-and-play feature also facilitates direct migration and application to other different video generation latent diffusion models (while ensuring the consistency of the latent space).

[0105] According to the test results, the video variational autoencoder in the method of the present invention is tested for video reconstruction on the test video data, and the peak signal-to-noise ratio and structural similarity are evaluated, achieving better performance than the existing method.

[0106] The present invention has the following beneficial effects: on the one hand, the unidirectional loop structure can realize the long-distance transmission of the historical timing information of the video; on the other hand, the introduction of the time domain residual module also strengthens the local timing information interaction between consecutive video frames, which can not only retain the original pre-trained image model structure, but also introduce three-dimensional operations at a lower cost; in this way, the network Rec3D-VAE in the method of the present invention combines the unidirectional loop structure with the three-dimensional operation, which can simultaneously strengthen the long-distance information transmission and local feature interaction during video decoding; in addition, the combination of spatial domain pre-training model loading and temporal layer fine-tuning strategy can also effectively alleviate the computational cost of video reconstruction model training, and efficiently train the video reconstruction model. Compared with the original single-frame decoding method, under the dual effects of network model design and training strategy, the method of the present invention can effectively improve the time domain jitter problem caused by the original single-frame decoding and alleviate the distortion phenomenon of the reconstruction result; the restored video spatial domain detail information is richer, the video picture is more natural and beautiful, and the visual quality of the decoded video is significantly improved. The quality of the reconstructed video can reach 33.22dB on PSNR and 0.8933 on SSIM, which exceeds other comparative methods.

[0107] When applying the video reconstruction method based on the video variational autoencoder provided by the present invention, it is not necessary to Figure 1 The steps are executed in the order shown. The specific execution order of the steps can be determined according to needs, and the present invention does not limit this.

[0108] The above is a video reconstruction method based on a video variational autoencoder provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding video reconstruction device based on a video variational autoencoder, such as Fig.12 shown.

[0109] Fig.12 A schematic diagram of a video reconstruction device based on a video variational autoencoder provided by the present invention, the device 1200 includes:

[0110] The construction module 1201 is used to introduce a time domain residual module on the basis of the original image decoder to form a pseudo three-dimensional residual module, and add a unidirectional loop module after the pseudo three-dimensional residual module to form a video decoding module; the video variational autoencoder network is formed by the original image encoder and the video decoding module; the video variational autoencoder network is trained to obtain a video variational autoencoder

[0111] The encoding module 1202 is used to input the video to be reconstructed into the original image encoder of the video variational autoencoder to obtain the encoding features of the video to be reconstructed in the low-dimensional latent space;

[0112] The decoding module 1203 is used to input the encoded features into the video decoding module of the video variational autoencoder, use the time domain residual module in the pseudo three-dimensional residual module to strengthen the time domain local feature interaction between consecutive video frames in the encoded features, and in the one-way loop module, transfer the time domain long-distance features of the video frames output by the pseudo three-dimensional residual module through the hidden state including the historical video frame features to obtain the reconstructed video.

[0113] For the specific limitations of the video reconstruction device based on the video variational autoencoder, please refer to the limitations of the video reconstruction method based on the video variational autoencoder above, which will not be repeated here. Each module in the above-mentioned video reconstruction device based on the video variational autoencoder can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0114] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The video reconstruction method based on video variational autoencoder is provided.

[0115] The present invention also provides Fig.13 The structural diagram of the computer device shown in FIG. Fig.13 As shown in the figure, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The video reconstruction method based on video variational autoencoder is provided.

[0116] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0117] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A video reconstruction method based on video variational autoencoder, characterized in that: include: A one-dimensional time-domain residual module is introduced on the basis of the original image decoder to form a pseudo three-dimensional residual module, and a one-way loop module is added after the pseudo three-dimensional residual module to form a video decoding module; A video variational autoencoder network is formed by using the original image encoder and the video decoding module; Training the video variational autoencoder network to obtain a video variational autoencoder; Inputting the video to be reconstructed into the original image encoder of the video variational autoencoder to obtain the encoding features of the video to be reconstructed in the low-dimensional latent space; The encoded features are input into the video decoding module of the video variational autoencoder, and the one-dimensional time domain residual module in the pseudo three-dimensional residual module is used to strengthen the time domain local feature interaction between consecutive video frames in the encoded features. In the one-way loop module, the video frames output by the pseudo three-dimensional residual module are transferred in the time domain long-distance features through the hidden state including the historical video frame features to obtain the reconstructed video.

2. The method according to claim 1, characterized in that The video decoding module includes multiple groups of pseudo three-dimensional residual modules and one-way circulation modules; each video frame of the video to be reconstructed corresponds to a group of pseudo three-dimensional residual modules and one-way circulation modules.

3. The method according to claim 2, characterized in that The pseudo three-dimensional residual module includes a two-dimensional spatial residual module and a one-dimensional temporal residual module; the one-dimensional temporal residual module in the pseudo three-dimensional residual module is used to enhance the temporal local feature interaction between consecutive video frames in the coding feature, including: For any video frame in the coding feature, the video frame and the frames before and after the video frame are input into the corresponding pseudo three-dimensional residual module, and the visual features of the video frame in the spatial domain are obtained through the two-dimensional spatial domain residual module; The visual features in the spatial domain are input into the one-dimensional temporal residual module to obtain temporal local interaction features after the video frame interacts with adjacent frames in the temporal domain.

4. The method according to claim 3, characterized in that The one-dimensional time domain residual module includes a plurality of time domain operation layers with skip connection branches; the visual features in the spatial domain are input into the one-dimensional time domain residual module to obtain the time domain local interaction features after the video frame interacts with the adjacent frames, including: The spatial visual features are sequentially input into the multiple temporal operation layers, and temporal local correlation features are obtained by strengthening the temporal local feature correlation between consecutive video frames; The visual features in the spatial domain are fused with the local correlation features in the temporal domain through skip connection branches to obtain the local interaction features in the temporal domain.

5. The method according to claim 4, characterized in that The channel flow of the time domain operation layer includes: group normalization, Swish activation, random inactivation layer and one-dimensional convolution in the time domain direction.

6. The method according to claim 2, characterized in that The input end of each unidirectional cycle module includes two parts, one part is connected to the output end of the corresponding pseudo three-dimensional residual module in the depth direction of the model, and the other part is connected to the hidden state output end of the unidirectional cycle module corresponding to the previous video frame in the time domain direction; the unidirectional cycle module includes a window mutual attention feature alignment unit, a convolution unit, a channel segmentation unit and a jump connection unit. In the unidirectional cycle module, the video frame output by the pseudo three-dimensional residual module is subjected to time domain long-distance feature transfer through the hidden state including the historical video frame feature to obtain a reconstructed video, including: For any video frame, the temporal local interaction feature of the video frame and the hidden state output by the one-way loop module corresponding to the previous video frame are input into the window mutual attention feature alignment unit to obtain an alignment feature; Performing three-channel splicing of the video temporal local interaction feature, the hidden state, and the alignment feature to obtain a spliced ​​feature, and fusing and transforming the spliced ​​feature through a convolution unit to obtain a fused feature; The fusion feature is divided along the feature channel dimension by a channel segmentation unit to obtain a hidden state residual part and a current frame feature residual part; The skip connection unit performs feature addition on the hidden state residual part and the hidden state output by the unidirectional cycle module corresponding to the previous video frame to obtain the hidden state output by the unidirectional cycle unit; The skip connection unit adds the feature residual of the current frame and the time-domain local interaction feature of the video frame to obtain the decoding feature output by the unidirectional cycle unit; The reconstructed video is obtained according to the decoding features output by each unidirectional cycle unit.

7. The method according to claim 6, characterized in that The window mutual attention feature alignment unit is established based on the shifted window mutual attention mechanism, and the window mutual attention feature alignment unit is formed by stacking two layers of window mutual attention modules; the temporal local interaction feature of the video frame and the hidden state output by the one-way loop module corresponding to the previous video frame are input into the mutual attention feature alignment unit to obtain the alignment feature, including: The temporal local interaction features of the video frame and the hidden state output by the unidirectional loop module corresponding to the previous video frame are group-normalized respectively, and the normalized temporal local interaction features and hidden state are window-divided; The temporal local interaction features and hidden states after window division are input into the first-layer window mutual attention module. Under each divided window, the temporal local interaction features of the video frame are used as query features, and the hidden state output by the one-way loop module corresponding to the previous video frame is used as the key and value of mutual attention. The multi-head mutual attention calculation is performed through the introduced sinusoidal position encoding mechanism. The result of the multi-head mutual attention calculation is feature fused with the temporal local interaction features of the video frame to obtain the alignment features of the first stage. The alignment features of the first stage and the windows of the hidden state are shifted, and the alignment features and hidden states after the window shift are input into the second-layer window mutual attention module. The second-layer window mutual attention module performs multi-head mutual attention calculation on the alignment features after the window shift and the hidden state, and the results of the multi-head mutual attention calculation are fused with the alignment features of the first stage to obtain the alignment features output by the window mutual attention feature alignment unit.

8. The method according to claim 2, characterized in that: Each group of pseudo three-dimensional residual modules and unidirectional loop modules are connected through a dual-branch upsampling module, and the dual-branch upsampling module is obtained by expanding the interpolation upsampling module after the spatial domain residual module of the original image decoder; the dual-branch upsampling module includes two branches, one branch includes the nearest neighbor difference and 2D convolution, and the other branch includes 2D convolution and pixel rearrangement, and the method further includes: For any video frame, the time-domain local interaction features output by the pseudo 3D residual module are sequentially subjected to nearest neighbor difference and 2D convolution to obtain a first up-sampled feature; The time-domain local interaction features output by the pseudo three-dimensional residual module are sequentially subjected to 2D convolution and pixel rearrangement to obtain a second up-sampled feature; The first up-sampled features and the second up-sampled features are fused, and the fused features are input into the one-way loop module.

9. The method according to claim 1, characterized in that: The video variational autoencoder includes multiple video decoding modules; the multiple video decoding modules are arranged in series, and the input of the next video decoding module is the output of the previous video decoding module.

10. The method according to claim 1, characterized in that The step of training the video variational autoencoder network to obtain a video variational autoencoder includes: A consistent initialization strategy is used to initialize the parameters of the video variational autoencoder network while maintaining the spatial domain pre-training knowledge to form a video reconstruction model; The video reconstruction model is fine-tuned in the time domain layer on the training video data to obtain the video variational autoencoder.

Images