A video reconstruction method based on video variational autoencoder

Abstract

The present invention discloses a video reconstruction method based on a video variational autoencoder, and relates to the field of video reconstruction technology. The method introduces a one-dimensional time-domain residual module on the basis of an original image decoder to form a pseudo-three-dimensional residual module. A one-way loop module is added after the pseudo-three-dimensional residual module to form a video decoding module. The video to be reconstructed is input into the original image encoder of the video variational autoencoder to obtain the coding features of the video to be reconstructed in a low-dimensional latent space. The coding features are input into the video decoding module of the video variational autoencoder, and the one-dimensional time-domain residual module in the pseudo-three-dimensional residual module is used to enhance the interaction of temporal local features between consecutive video frames in the coding features. In the one-way loop module, the video frames output by the pseudo-three-dimensional residual module are subjected to temporal long-distance feature transfer through hidden states including historical video frame features to obtain a reconstructed video. This method can improve the visual quality of the reconstructed video.

Description

Technical Field

[0001] The present invention relates to the technical field of video reconstruction, and in particular to a video reconstruction method based on a video variational autoencoder. Background Art

[0002] In recent years, artificial intelligence content generation (AIGC) technology combined with large models has developed rapidly in academia and industry. However, while large models bring performance advantages, they also put tremendous pressure on computing power and resources, becoming a major challenge for the practical application of AIGC. In the field of image generation, latent diffusion generative models (LDMs) use variational autoencoders to map pixel space to latent space, thereby converting the original diffusion generation process that needs to be carried out in high-dimensional space to low-dimensional space, which greatly reduces the amount of computation required for the actual application of the diffusion model. Therefore, latent diffusion generative models represented by Stable Diffusion have become another new paradigm for visual content generation algorithms.

[0003] In existing technologies, compared to image generation tasks, video generation tasks require larger networks, more data, and a longer training process. In reality, they often face the dilemma of limited resources and a lack of video data. Therefore, video generation models often use existing image generation models as pre-training models. This not only facilitates network design, but also provides a foundation for training video generation models, thereby accelerating training and, to a certain extent, alleviating the dependence on data volume. Given the success of latent diffusion generative models in image generation tasks, some research works have also attempted to apply latent diffusion to video generation, directly using existing image latent diffusion models as pre-training models. In order to utilize the latent space pre-trained for image generation, the video generation process also needs to use the same encoder for spatial mapping. These methods usually use the original image autoencoder to encode or decode the video to achieve video reconstruction.

[0004] However, the independent decoding method of each frame in the existing technology ignores the temporal correlation between video frames, resulting in poor temporal continuity of the decoding results, introducing flickering, jitter and distortion, and ultimately affecting the quality of the generated video. Summary of the Invention

[0005] Based on this, it is necessary to provide a video reconstruction method based on video variational autoencoder to address the above technical problems. This method can improve the visual degradation phenomenon caused by directly using pre-trained image variational autoencoders in video reconstruction and video generation tasks, alleviate the time domain jitter problem caused by the original single-frame decoding, and improve the visual quality of the reconstructed video.

[0006] The present invention adopts the following technical solutions:

[0007] The present invention provides a video reconstruction method based on a video variational autoencoder, comprising:

[0008] A one-dimensional time-domain residual module is introduced on the basis of the original image decoder to form a pseudo three-dimensional residual module. A one-way loop module is added after the pseudo three-dimensional residual module to form a video decoding module.

[0009] The video variational autoencoder network is constructed by the original image encoder and video decoding module;

[0010] Train the video variational autoencoder network to obtain a video variational autoencoder;

[0011] Input the video to be reconstructed into the original image encoder of the video variational autoencoder to obtain the encoding features of the video to be reconstructed in the low-dimensional latent space;

[0012] The encoded features are input into the video decoding module of the video variational autoencoder, and the one-dimensional time domain residual module in the pseudo three-dimensional residual module is used to enhance the interaction of temporal local features between consecutive video frames in the encoded features. In the unidirectional loop module, the video frames output by the pseudo three-dimensional residual module are transferred through the hidden state including the historical video frame features to obtain the reconstructed video.

[0013] Preferably, the video decoding module includes multiple groups of pseudo 3D residual modules and unidirectional loop modules; each video frame of the video to be reconstructed corresponds to a group of pseudo 3D residual modules and unidirectional loop modules.

[0014] Preferably, the pseudo 3D residual module includes a 2D spatial residual module and a 1D temporal residual module; and utilizing the 1D temporal residual module in the pseudo 3D residual module to enhance the temporal local feature interaction between consecutive video frames in the coding features includes:

[0015] For any video frame in the coding feature, the video frame and the frames before and after the video frame are input into the corresponding pseudo 3D residual module, and the visual features of the video frame in the spatial domain are obtained through the 2D spatial residual module;

[0016] The visual features in the spatial domain are input into the one-dimensional temporal residual module to obtain the temporal local interaction features after the video frame interacts with the adjacent frames in the temporal domain.

[0017] Preferably, the one-dimensional time domain residual module includes multiple time domain operation layers with skip connection branches; the visual features in the spatial domain are input to the one-dimensional time domain residual module to obtain the time domain local interaction features after the video frame interacts with the adjacent frames, including:

[0018] The spatial visual features are sequentially input into multiple temporal operation layers, and the temporal local correlation features are obtained by strengthening the temporal local feature correlation between consecutive video frames.

[0019] The visual features in the spatial domain are fused with the local correlation features in the temporal domain through skip connection branches to obtain the local interaction features in the temporal domain.

[0020] Preferably, the channel process of the time domain operation layer includes: group normalization, Swish activation, random inactivation layer and one-dimensional convolution in the time domain direction.

[0021] Preferably, the input end of each unidirectional cycle module includes two parts, one part is connected to the output end of the corresponding pseudo 3D residual module in the depth direction of the model, and the other part is connected to the hidden state output end of the unidirectional cycle module corresponding to the previous video frame in the time domain direction; the unidirectional cycle module includes a window mutual attention feature alignment unit, a convolution unit, a channel splitting unit and a jump connection unit. In the unidirectional cycle module, the video frame output by the pseudo 3D residual module is subjected to time domain long-distance feature transfer through the hidden state including the historical video frame features to obtain a reconstructed video, including:

[0022] For any video frame, the temporal local interaction features of the video frame and the hidden state output by the unidirectional loop module corresponding to the previous video frame are input into the window mutual attention feature alignment unit to obtain the aligned features;

[0023] The video temporal local interaction features, hidden state and alignment features are spliced ​​in three channels to obtain spliced ​​features, and the spliced ​​features are fused and transformed through convolution units to obtain fused features;

[0024] The fusion feature is divided along the feature channel dimension through the channel segmentation unit to obtain the hidden state residual part and the current frame feature residual part;

[0025] The hidden state residual part is added to the hidden state output by the unidirectional cycle module corresponding to the previous video frame through the jump connection unit to obtain the hidden state output by the unidirectional cycle unit;

[0026] The feature residual of the current frame and the temporal local interaction features of the video frame are added together through the skip connection unit to obtain the decoding features output by the unidirectional recurrent unit.

[0027] The reconstructed video is obtained based on the decoded features output by each unidirectional cycle unit.

[0028] Preferably, the window mutual attention feature alignment unit is established based on the shifted window mutual attention mechanism, and the window mutual attention feature alignment unit is composed of two layers of window mutual attention modules stacked together; the temporal local interaction features of the video frame and the hidden state output by the unidirectional loop module corresponding to the previous video frame are input into the window mutual attention feature alignment unit to obtain the alignment features, including:

[0029] The temporal local interaction features of the video frame and the hidden state output by the unidirectional loop module corresponding to the previous video frame are group-normalized, and the normalized temporal local interaction features and hidden state are windowed;

[0030] The temporal local interaction features and hidden states after window division are input into the first-layer window mutual attention module. Under each divided window, the temporal local interaction features of the video frame are used as query features, and the hidden state output by the one-way recurrent module corresponding to the previous video frame is used as the key and value of the mutual attention. Multi-head mutual attention is calculated through the introduced sinusoidal position encoding mechanism. The results of the multi-head mutual attention calculation are fused with the temporal local interaction features of the video frame to obtain the first-stage alignment features.

[0031] The windows of the aligned features and hidden states in the first stage are shifted, and the aligned features and hidden states after the window shift are input into the second-layer window mutual attention module. The second-layer window mutual attention module performs multi-head mutual attention calculation on the aligned features and hidden states after the window shift, and the results of the multi-head mutual attention calculation are fused with the aligned features in the first stage to obtain the aligned features output by the window mutual attention feature alignment unit.

[0032] Preferably, each group of pseudo three-dimensional residual modules and unidirectional cyclic modules are connected via a dual-branch upsampling module, which is obtained by expanding the interpolation upsampling module after the spatial residual module of the original image decoder; the dual-branch upsampling module includes two branches, one branch includes nearest neighbor difference and 2D convolution, and the other branch includes 2D convolution and pixel rearrangement. The method further includes:

[0033] For any video frame, the temporal local interaction features output by the pseudo 3D residual module are sequentially subjected to the nearest neighbor difference and 2D convolution to obtain the first up-sampled features;

[0034] The temporal local interaction features output by the pseudo 3D residual module are sequentially subjected to 2D convolution and pixel rearrangement to obtain the second up-sampled features;

[0035] The first up-sampled feature and the second up-sampled feature are fused, and the fused feature is input into the unidirectional loop module.

[0036] Preferably, the video variational autoencoder includes multiple video decoding modules; the multiple video decoding modules are arranged in series, and the input of the next video decoding module is the output of the previous video decoding module.

[0037] Preferably, training the video variational autoencoder network to obtain a video variational autoencoder includes:

[0038] A consistent initialization strategy is used to initialize the parameters of the video variational autoencoder network while maintaining the spatial domain pre-training knowledge to form a video reconstruction model.

[0039] The video reconstruction model is fine-tuned in the time domain layer on the training video data to obtain a video variational autoencoder.

[0040] The present invention provides a video reconstruction device based on a video variational autoencoder, comprising:

[0041] A construction module is used to introduce a time domain residual module on the basis of the original image decoder to form a pseudo three-dimensional residual module, and add a one-way loop module after the pseudo three-dimensional residual module to form a video decoding module; a video variational autoencoder network is formed by the original image encoder and the video decoding module; the video variational autoencoder network is trained to obtain a video variational autoencoder.

[0042] The encoding module is used to input the video to be reconstructed into the original image encoder of the video variational autoencoder to obtain the encoding features of the video to be reconstructed in the low-dimensional latent space;

[0043] The decoding module is used to input the encoded features into the video decoding module of the video variational autoencoder, use the time domain residual module in the pseudo three-dimensional residual module to strengthen the time domain local feature interaction between consecutive video frames in the encoded features, and in the one-way loop module, transfer the time domain long-distance features of the video frames output by the pseudo three-dimensional residual module through the hidden state including the historical video frame features to obtain the reconstructed video.

[0044] The present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the video reconstruction method based on the video variational autoencoder is implemented.

[0045] The present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the video reconstruction method based on the video variational autoencoder is implemented.

[0046] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0047] In the present invention, a time domain residual module is first introduced on the basis of the original image decoder to form a pseudo three-dimensional residual module, and a one-way loop module is added after the pseudo three-dimensional residual module to form a video decoding module. In this way, when decoding the coding features of the reconstructed video, the video variational autoencoder can simultaneously consider the time change information between adjacent frames, thereby realizing the interaction of short-distance features in the time domain between adjacent video frames; and, when using the hidden state to transfer long-distance features in the time domain to each coding feature, the hidden state includes the features of the historical video frames. In this way, by transferring the features of the historical video frames to the current video frame, the continuous process of the video can be better described, the continuity of the reconstructed video is improved, and the time domain jitter problem caused by the original single frame decoding is effectively improved, thereby improving the visual quality of the reconstructed video. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0049] Figure 1 A schematic flow chart of a video reconstruction method based on a video variational autoencoder provided by the present invention;

[0050] Figure 2 A schematic diagram of the structure of a video variational autoencoder provided by the present invention;

[0051] Figure 3 A schematic structural diagram of a video decoding unit provided by the present invention;

[0052] Figure 4 A schematic diagram of the structure of a two-dimensional spatial domain residual module and a one-dimensional temporal domain residual module provided by the present invention;

[0053] Figure 5 A schematic structural diagram of a one-way circulation module provided by the present invention;

[0054] Figure 6 A schematic diagram of the structure of a window mutual attention feature alignment unit provided by the present invention;

[0055] Figure 7 A schematic diagram of the structure of a dual-branch upsampling module and a dual-branch sampling module at a video decoder provided by the present invention;

[0056] Figure 8 A schematic diagram showing some visualization results of the video variational autoencoder Rec3D-VAE provided by the present invention and four other comparison models on a test set;

[0057] Figure 9Schematic diagram of some visualization results of another video variational autoencoder Rec3D-VAE provided by the present invention and four other comparison models;

[0058] Figure 10 Schematic diagram of some visualization results of another video variational autoencoder Rec3D-VAE provided by the present invention and four other comparison models on the test set;

[0059] Figure 11 A schematic diagram of the application effect of a video variational autoencoder Rec3D-VAE provided by the present invention in the Hotshot-XL latent diffusion video generation model;

[0060] Figure 12 A schematic diagram of a video reconstruction device based on a video variational autoencoder provided by the present invention;

[0061] Figure 13 A schematic diagram of a computer device for implementing a video reconstruction method based on a video variational autoencoder provided by the present invention. DETAILED DESCRIPTION

[0062] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present invention and corresponding drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0063] Currently, researchers have noticed the visual degradation caused by applying image variational autoencoders to videos and are attempting to improve the visual quality of generated videos by proposing a post-processing model called Video Deflicking (VD). However, the neural network filtering (NF) method they proposed requires separate online training for each video to be reconstructed, and the addition of additional models also complicates the video generation process. Some methods have also attempted to introduce time-domain operation layers (such as adding time-domain convolutional layers or attention layers) into the original image decoder model and fine-tune the video data (Fine-Tune) to construct a video decoder. Although this simple method of introducing model layers is simple, its effectiveness in enhancing video quality still needs to be improved.

[0064] Based on this, the present invention provides a video reconstruction method based on video variational autoencoder, which can improve the visual degradation phenomenon caused by directly using pre-trained image variational autoencoders in video reconstruction and video generation tasks, alleviate the temporal jitter problem caused by the original single-frame decoding, and improve the visual quality of the decoded video.

[0065] The technical solutions provided by various embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0066] Figure 1 The following is a flow chart of a video reconstruction method based on a video variational autoencoder in the present invention, which specifically includes the following steps:

[0067] S101, based on the original image decoder, a time domain residual module is introduced to form a pseudo three-dimensional residual module, and a unidirectional loop module is added after the pseudo three-dimensional residual module to form a video decoding module; a video variational autoencoder network is formed by the original image encoder and the video decoding module; the video variational autoencoder network is trained to obtain a video variational autoencoder.

[0068] Among them, the video decoding module includes multiple groups of pseudo three-dimensional residual modules and one-way loop modules; each video frame of the video to be reconstructed corresponds to a group of pseudo three-dimensional residual modules and one-way loop modules. Each group of pseudo three-dimensional residual modules and one-way loop modules can be regarded as a video decoding unit, and the video variational autoencoder includes multiple video decoding modules; multiple video decoding modules are arranged in series, and the input of the next video decoding module is the output of the previous video decoding module; specifically, the input of the pseudo three-dimensional residual module in the first video decoding module is the output of the original image encoder, and the input of the pseudo three-dimensional residual modules in subsequent video decoding modules are all the output of the one-way loop module in the previous video decoding module, and the one-way loop module of the last video decoding module outputs the reconstructed video. As Figure 2 As shown, Figure 2 Schematic diagram of the structure of a video variational autoencoder; multiple video decoding modules constitute the video decoder of the video variational autoencoder.

[0069] Specifically, each group of pseudo 3D residual modules and unidirectional loop modules can be regarded as a video decoding unit. Each video decoding unit also includes a dual-branch upsampling module. The input end of each pseudo 3D residual module is connected to the input end of the dual-branch upsampling module. The input end of each unidirectional loop module is respectively connected to the output end of the dual-branch upsampling module of the decoding unit to which it belongs, and the hidden state output end of the unidirectional loop module of the previous video frame. The stacking of multiple unidirectional loop modules in the time domain direction constitutes a unidirectional loop structure to realize long-distance feature transmission in the video time domain, such as Figure 3 As shown, Figure 3 The figure is a structural diagram of a video decoding unit.

[0070] Among them, the pseudo three-dimensional residual module includes the original two-dimensional spatial residual module and the one-dimensional time domain residual module designed by the present invention, the input end of the two-dimensional spatial residual module is connected to the output end of the original image encoder, the output end of the two-dimensional spatial residual module is connected to the input end of the one-dimensional time domain residual module, and the output end of the one-dimensional time domain residual module is connected to the feature input end of the unidirectional loop module; Figure 4 As shown, Figure 4 It is a structural diagram of the two-dimensional spatial domain residual module and the one-dimensional time domain residual module; Figure 4 Figure (a) is the structural diagram of the two-dimensional spatial residual module. Figure 4 Figure (b) shows the structure of the one-dimensional temporal residual module. The two-dimensional spatial residual module is composed of a stack of spatial operation layers with skip connections. The channel flow of each spatial operation layer includes group normalization, Swish activation, and spatial 2D convolution. The one-dimensional temporal residual module calculates local correlation features of consecutive video frames in the temporal direction. It is composed of four stacked temporal operation layers with skip connections. The channel flow of each temporal operation layer includes group normalization, Swish activation, random dropout layer, and one-dimensional convolution in the temporal direction.

[0071] The unidirectional recurrent structure uses hidden states to perform long-distance feature transfer in the video time domain and is composed of multiple stacked unidirectional recurrent modules; the unidirectional recurrent module includes a window mutual attention feature alignment unit, a convolution unit, a channel splitting unit, and a jump connection unit.

[0072] Take the i-th decoding stage of the decoder (or called the i-th video decoding module) as an example, Figure 5 As shown, represents the feature of the t-1th frame output by the pseudo 3D residual module at the i-th decoding stage, F t i represents the features of the t-th frame output by the pseudo 3D residual module at the i-th decoding stage, Represents the features of the t+1th frame output by the pseudo 3D residual module at the i-th decoding stage, in F t i In the corresponding one-way loop module, according to F t i And the hidden features output by the one-way loop module corresponding to the t-1th frame Output F t i+1 as well as Specifically, the implementation process of each one-way loop module includes: when operating the features of the t-th frame of the video, the one-way loop module not only obtains the current feature F t i , and also obtain the hidden state of the output of the previous one-way loop module operation As input; after the operation of this stage, the feature F of the current time position is output t i+1 As the input of stage i+1, it outputs As the hidden state input of the next unidirectional loop module in the same stage; thus, each video frame feature that has been calculated is accumulated into the hidden state and transmitted in the time dimension as the calculation deepens.

[0073] Inside the unidirectional loop module, the window mutual attention feature alignment unit based on the shifted window cross attention mechanism (SWCA) in SwinTransformer converts the hidden state Align to the current frame feature F t i , get the aligned hidden state Then the current frame state F t i , aligned hidden states and the original hidden state Splicing along the channel latitude, and performing feature fusion and transformation through the convolution unit. When output from the convolution unit, the number of channels is transformed to twice the number of channels of the original single frame input. Then, the channel splitting unit divides the feature map along the feature channel dimension to obtain the hidden state residual part and the current frame feature residual part. The two residual parts are added to the corresponding skip connection features to obtain the feature output F at this stage and time position. t i+1 and hidden state

[0074] The window mutual attention feature alignment unit is composed of two layers of window mutual attention stacked together, such as Figure 6 As shown in FIG, in the first layer of window mutual attention, the window is divided and then the multi-head mutual attention calculation is performed. In the second layer of window mutual attention, the window shift operation is performed and then the multi-head mutual attention calculation is performed. In addition, in order to alleviate the problem of missing original two-dimensional position information during visual attention calculation, the present invention additionally introduces sinusoidal positional encoding (PE) during the original window mutual attention calculation to provide the original image two-dimensional information. Specifically, the current frame feature F is used in the calculation. t i As the query feature (Query, represented by Q in the figure), the historical hidden state As the key (Key, represented by K in the figure) and value (Value, represented by V in the figure); additional sinusoidal position encoding is introduced in the original window mutual attention operation to provide the original two-dimensional image information; based on the regional similarity between the hidden state and the current frame features, the features of the corresponding position of the hidden state are "moved" and "weighted", aligned to the current features, and the aligned hidden state features are output

[0075] It should be noted that the decoding module itself contains a sufficient number of computational layers and does not use the multi-layer linear perceptron in the original SwinTransformer.

[0076] The interpolation upsampling module after the original two-dimensional spatial residual module is expanded to form a dual-branch upsampling module, such as Figure 7 As shown in the figure, the dual-branch upsampling module adds a parallel branch based on the original nearest neighbor interpolation and combines it with pixel realignment in the super-resolution domain to form a dual branch. This enriches the features of decoding upsampling and optimizes decoding quality. The dual-branch upsampling module consists of two parallel branches: one branch first performs nearest neighbor interpolation and then 2D convolution, and the other branch first performs 2D convolution and then pixel realignment in the super-resolution domain. Finally, the results of the two branches are fused.

[0077] It should be noted that Figure 7 The figure shows the connection between multiple video decoding modules, and the video decoding module includes a pseudo three-dimensional residual module and a dual-branch upsampling module. The one-way loop module is not illustrated. In actual applications, the dual-branch upsampling module in the previous video decoding module and the pseudo three-dimensional residual module in the next video decoding module are connected through the one-way loop module in the previous video decoding module.

[0078] Based on the above-mentioned video variational autoencoder network structure, the video variational autoencoder network is trained to obtain a video variational autoencoder. In an exemplary embodiment, the video variational autoencoder network is trained to obtain a video variational autoencoder, including: using a consistent initialization strategy to initialize the parameters of the video variational autoencoder network while maintaining spatial domain pre-training knowledge to form a video reconstruction model; performing temporal layer fine-tuning training on the video reconstruction model on the training video data to obtain a video variational autoencoder.

[0079] To load and maintain the spatial pre-trained knowledge in the video decoder, a special initialization method is required: when initializing the video reconstruction model, identity initialization is used to ensure that the output of the video model is identical to the result of a single-frame operation of the original image model. Specifically, after the original image decoder parameters are fully loaded into the 2D module, the unidirectional recurrent module in the video reconstruction model and the one-dimensional temporal residual module in the pseudo-3D module are initialized. Due to the skip connection design, only the parameters of the last fused convolution layer in the unidirectional recurrent module, the last convolution layer in the one-dimensional residual module, and the convolution parameters of the pixel rearrangement branch need to be initialized to zero; all other parameters are randomly initialized. Regarding the choice of pre-trained image reconstruction model, the variational autoencoder within Stable Diffusion XL (SDXL), a high-quality new version of the Stable Diffusion series of Vincent Image Large Models, can be used as the pre-trained image variational autoencoder.

[0080] In order to train and test the video reconstruction model, this embodiment constructs a large-scale, high-quality video dataset in an open domain. The specific process is as follows: first, the material website Pexels is selected as the original data source, and the videos on the site are downloaded in batches. The videos on this material website are characterized by high resolution and high quality; then, the video automatic reading tool (FFmpeg) is used to read these original videos one by one and filter out videos that cannot be read normally; finally, long videos are cut to balance the data.

[0081] The video dataset constructed in this embodiment contains 345K video data, of which 1K are used to constitute the test dataset, and the remaining 344K videos constitute the training dataset; this dataset is high-quality, large-scale, and open-domain oriented.

[0082] The video reconstruction model is fine-tuned on the time domain layer on the training video data. Specifically, in terms of training strategy, in order to maintain the original pre-trained image model knowledge, this embodiment only fine-tunes the newly added time domain operation layer, while fixing all two-dimensional pre-training layers; in addition, since the original latent space needs to be kept unchanged, the encoding module will not participate in the training.

[0083] The loss function for fine-tuning the video reconstruction model in this embodiment consists of a reconstruction loss and a perceptual loss based on VGG16. Since the encoder does not need to be trained, the KL divergence loss term used in general variational autoencoder training is not required in this embodiment. However, reparameterized sampling of the encoded output of the video reconstruction model is still required.

[0084] After the video variational autoencoder is constructed, the video to be reconstructed is input into the encoder of the video variational autoencoder to obtain the encoding features.

[0085] S102: Input the video to be reconstructed into the original image encoder of the video variational autoencoder to obtain the coding features of the video to be reconstructed in the low-dimensional latent space.

[0086] Specifically, see Figure 2 The image encoder includes multiple encoding modules, each encoding module corresponds to a video frame, and the video to be reconstructed is input into the original image encoder of the video variational autoencoder. Each encoding module encodes each video frame in the video to be reconstructed respectively to obtain the encoding features of the video to be reconstructed in the low-dimensional latent space.

[0087] S103, input the coding features into the video decoding module of the video variational autoencoder, use the one-dimensional time domain residual module in the pseudo three-dimensional residual module to strengthen the time domain local feature interaction between consecutive video frames in the coding features, and in the one-way loop module, transfer the time domain long-distance features of the video frames output by the pseudo three-dimensional residual module through the hidden state including the historical video frame features to obtain the reconstructed video.

[0088] Please refer to Figure 2 When the original image encoder inputs the coding features into the video decoding module, the pseudo three-dimensional residual module of each group of the video decoding module receives the coding features of the corresponding video frame and the adjacent frames of the video frame respectively; for example, if the video frame is the first frame, the adjacent video frame of the video frame can be the second frame; if the video frame is the last frame, the adjacent frame of the video frame is the previous frame of the last frame; if the video frame is an intermediate frame, the adjacent frames of the video frame are the previous frame and the next frame of the video frame.

[0089] See Figure 4 , using the one-dimensional time domain residual module in the pseudo three-dimensional residual module to strengthen the time domain local feature interaction between consecutive video frames in the coding features, including: for any video frame in the coding features, the video frame and the previous and next frames of the video frame are input into the corresponding pseudo three-dimensional residual module, and the visual features of the video frame in the spatial domain are obtained through the two-dimensional spatial domain residual module; the visual features in the spatial domain are input into the one-dimensional time domain residual module, and the time domain local interaction features after the video frame interacts with the adjacent frames in the time domain are obtained.

[0090] The video frame and the preceding and following frames of the video frame are input into the corresponding pseudo three-dimensional residual module, and the visual features of the video frame in the spatial domain are obtained through the two-dimensional spatial residual module, including: the video frame and the preceding and following frames of the video frame are input into multiple spatial operation layers in sequence, and feature fusion is performed through the residual structure to obtain the visual features of the video frame in the spatial domain; the visual features in the spatial domain are input into the one-dimensional time domain residual module to obtain the time domain local interaction features after the video frame interacts with the adjacent frames, including: the visual features in the spatial domain are input into multiple time domain operation layers in sequence, and the time domain local correlation features are obtained by strengthening the time domain local feature association between consecutive video frames; the visual features in the spatial domain are fused with the time domain local correlation features through skip connection branches to obtain the time domain local interaction features.

[0091] Optionally, the input of each unidirectional loop module includes two parts, one part is connected to the output of the corresponding pseudo 3D residual module in the depth direction of the model, and the other part is connected to the hidden state output of the unidirectional loop module corresponding to the previous video frame in the time domain direction; please continue to refer to Figure 5 In the unidirectional cycle module, the video frame output by the pseudo three-dimensional residual module is subjected to time domain long-distance feature transfer through the hidden state including the historical video frame features to obtain a reconstructed video, including: for any video frame, the time domain local interaction features of the video frame and the hidden state output by the unidirectional cycle module corresponding to the previous video frame are input into the window mutual attention feature alignment unit to obtain an alignment feature; the video time domain local interaction features, hidden state and alignment features are three-channel spliced ​​to obtain a spliced ​​feature, and the spliced ​​features are fused and transformed through a convolution unit to obtain a fused feature; the fused feature is feature-divided along the feature channel dimension by a channel splitting unit to obtain a hidden state residual part and a current frame feature residual part; the hidden state residual part is feature-added with the hidden state output by the unidirectional cycle module corresponding to the previous video frame through a jump connection unit to obtain a hidden state output by the unidirectional cycle unit; the current frame feature residual part and the time domain local interaction features of the video frame are feature-added through a jump connection unit to obtain a decoding feature of the video frame output by the unidirectional cycle unit; and a reconstructed video is obtained based on the decoding features output by each unidirectional cycle unit.

[0092] Specifically, the current frame features that have undergone time-domain local feature interaction and the historical hidden state output by the unidirectional loop module of the previous video frame are input into the window mutual attention feature alignment unit to obtain the aligned features; the current frame features, hidden state and aligned features are spliced ​​along the channel dimension, and the spliced ​​features are fused and transformed through the convolution unit so that the current frame features and the historical state features fully interact to obtain the fused features; the fused features are feature-divided along the feature channel dimension by the channel splitting unit to obtain the hidden state residual part and the current frame feature residual part; the hidden state residual part is feature-added with the input historical hidden state by the jump connection unit to obtain the hidden state output at this stage and this time position; the current frame feature residual part is feature-added with the current frame feature by the jump connection unit to obtain the coding features of the current video frame that fully refer to the historical frame information.

[0093] For details, please see Figure 5 As shown, the local interaction feature of the video time domain is F t i , the hidden state output by the one-way loop module of the previous video frame is Alignment feature is The hidden state output at this stage and time position is The decoding characteristics of the current video frame with full reference to the historical frame information are

[0094] Optionally, the window mutual attention feature alignment unit is established based on the shifted window mutual attention mechanism. The window mutual attention feature alignment unit is composed of two layers of window mutual attention modules stacked together. Please continue to refer to Figure 6, the temporal local interaction features of the video frame and the hidden state output by the one-way cycle module corresponding to the previous video frame are input into the window mutual attention feature alignment unit to obtain the alignment features, including: group normalizing the temporal local interaction features of the video frame and the hidden state output by the one-way cycle module corresponding to the previous video frame, and dividing the normalized temporal local interaction features and hidden state into windows; inputting the temporal local interaction features and hidden state after dividing the windows into the first layer of the window mutual attention module, and under each divided window, using the temporal local interaction features of the video frame as the query feature, and using the hidden state output by the one-way cycle module corresponding to the previous video frame as the query feature. The multi-head mutual attention is calculated by the key and value of mutual attention, and the sinusoidal position encoding mechanism is introduced; the result of the multi-head mutual attention calculation is fused with the time-domain local interaction feature of the video frame to obtain the alignment feature of the first stage; the alignment feature of the first stage and the window of the hidden state are shifted, and the alignment feature and hidden state after the window shift are input into the second-layer window mutual attention module, and the multi-head mutual attention calculation is performed on the alignment feature and hidden state after the window shift through the second-layer window mutual attention module, and the result of the multi-head mutual attention calculation is fused with the alignment feature of the first stage to obtain the alignment feature output by the window mutual attention feature alignment unit.

[0095] In an exemplary embodiment, see Figure 3 and Figure 7 As shown, each group of pseudo three-dimensional residual modules and unidirectional loop modules are connected through a dual-branch upsampling module, and the dual-branch upsampling module is obtained by expanding the interpolation upsampling module after the spatial residual module of the original image decoder; the dual-branch upsampling module includes two branches, one branch includes nearest neighbor difference and 2D convolution, and the other branch includes 2D convolution and pixel rearrangement. This embodiment includes: for any video frame, the time domain local interaction features output by the pseudo three-dimensional residual module are sequentially subjected to nearest neighbor difference and 2D convolution to obtain a first upsampling feature; the time domain local interaction features output by the pseudo three-dimensional residual module are sequentially subjected to 2D convolution and pixel rearrangement to obtain a second upsampling feature; the first upsampling feature and the second upsampling feature are fused, and the fused feature is input into the unidirectional loop module.

[0096] That is, the video time-domain interaction features input into the one-way loop module are actually the fusion features output by the dual-branch upsampling module, which will not be described in detail in this embodiment.

[0097] The above describes the construction and application process of the video variational autoencoder. However, in actual applications, after the video variational autoencoder is constructed, the performance of the video variational autoencoder can be evaluated first. After the video variational autoencoder meets the performance requirements, the video can be reconstructed through the video variational autoencoder.

[0098] It should be noted that the video decoding module in the video variational autoencoder in the present invention is expanded on the basis of the original image decoder, and the video variational autoencoder should also include other parts of the original image decoder, but the embodiments of the present application will not be repeated here. The embodiments of the present invention only explain the decoding process involved in the present invention. After a detailed description of the time domain residual module and the unidirectional loop module, those skilled in the art should be able to understand the meaning of the expansion based on the original image decoder mentioned in the present invention and the necessary steps that the video decoder of the present invention should include.

[0099] In an exemplary embodiment, the final trained video variational autoencoder (Recurrent 3D Video VAE, Rec3D-VAE) is evaluated using a test set, using the evaluation indicators: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM), and compared with other methods, as shown in Table 1. Table 1 shows the video reconstruction results of different variational autoencoders on the test set.

[0100] Table 1

[0101] SD-V2.1-VAE SDXL-VAE SVD-VAE VidRD-VAE Rec3D-VAE PSNR 32.19 32.73 31.35 32.24 33.22 SSIM 0.8717 0.8846 0.8756 0.8728 0.8933

[0102] Please refer to the test results in Table 1. Thanks to the method that combines the unidirectional recurrent structure with three-dimensional operations, and strengthens the global and local information interaction in the video time domain, the method of the present invention, Rec3D-VAE, surpasses the other four comparison models in PSNR and SSIM. The video reconstruction result reaches 33.22dB in PSNR and 0.8933 in SSIM.

[0103] Next, in this embodiment, visualization is used to demonstrate the effectiveness and superiority of the video variational autoencoder provided by the present invention in the task of decoding and reconstructing videos. Figure 8-10As shown, some visualization results of the video variational autoencoder Rec3D-VAE constructed by the present invention and the other four comparison models on the test set are shown. Among them, Input is marked with the original input, SD-V2.1-VAE (Stable Diffusion Version 2.1-Variational Autoencode), SDXL-VAE (Stable Diffusion XL-Variational Autoencode), SVD-VAE (Stable Video Diffusion-Variational Autoencode) and VidRD-VAE (Video Reuse and Diffuse-Variational Autoencode) are respectively marked with the results of the four methods compared in Table 1 above, and Rec3D-VAE is marked with the result of the video reconstruction method of the video variational autoencoder provided by the present invention. In some results, arrows are also used to mark the areas that need to be compared. From the visualization results, the video variational autoencoder Rec3D-VAE provided by the present invention has obvious advantages in the reconstruction effect of details. Figure 8 The advantages of the method of the present invention (represented by Rec3D-VAE in the figure) in restoring clear and detailed information of images are demonstrated; Figure 9 The advantages of the method of the present invention (represented by Rec3D-VAE in the figure) in alleviating the distortion of the reconstruction results are demonstrated. In addition, unlike the distortion that often exists in the results of other methods, the results of the method of the present invention are more natural and beautiful in overall visual appearance ( Figure 10 ), which is due to the fact that the method of the present invention simultaneously strengthens the long-distance information transmission and local feature interaction during video decoding.

[0104] The video variational autoencoder proposed in the present invention is ultimately aimed at the video generation task based on the latent diffusion generative model. In another embodiment of the present invention, the existing video generation model is combined to illustrate the role of the video variational autoencoder Rec3D-VAE proposed in the present invention in improving the quality of generated videos. In this example, the open source video generation model Hotshot-XL based on SDXL is used as the generation model in the latent space, and the video variational autoencoder proposed in the present invention is used to replace the autoencoder inside Hotshot-XL to decode and map the generated data in the latent space to the pixel space. Please refer to Figure 11 As shown in the figure, for the same generated data in the latent space, compared with the video results directly generated by SDXL-VAE inside Hotshot-XL (see Appendix Figure 11 The first row, marked with a red box), after being decoded by the Rec3D-VAE proposed in this invention (see Appendix Figure 11The second row (marked by the yellow box) shows richer textures and more complete outlines, significantly improving the visual quality of the generated video. This demonstrates the effectiveness of our improved method for video generation tasks. Its plug-and-play nature also facilitates direct migration to other latent diffusion models for video generation (while ensuring the consistency of the latent space).

[0105] According to the test results, the video variational autoencoder in the method of the present invention was tested for video reconstruction on the test video data, and the peak signal-to-noise ratio and structural similarity were evaluated, achieving better performance than the existing method.

[0106] The present invention has the following beneficial effects: on the one hand, the unidirectional loop structure can realize the long-distance transmission of the historical temporal information of the video; on the other hand, the introduction of the time domain residual module also strengthens the local temporal information interaction between consecutive video frames, which not only retains the original pre-trained image model structure, but also introduces three-dimensional operations at a lower cost; in this way, the network Rec3D-VAE in the method of the present invention combines the unidirectional loop structure with the three-dimensional operation, which can simultaneously strengthen the long-distance information transmission and local feature interaction during video decoding; in addition, the combination of spatial domain pre-training model loading and temporal layer fine-tuning strategy can also effectively alleviate the computational cost of video reconstruction model training, and efficiently train the video reconstruction model. Compared with the original single-frame decoding method, under the dual effects of network model design and training strategy, the method of the present invention can effectively improve the time domain jitter problem caused by the original single-frame decoding and alleviate the distortion phenomenon of the reconstruction result; the recovered video spatial domain detail information is richer, the video picture is more natural and beautiful, and the visual quality of the decoded video is significantly improved. The quality of the reconstructed video can reach 33.22dB in PSNR and 0.8933 in SSIM, which exceeds other comparison methods.

[0107] When applying the video reconstruction method based on video variational autoencoder provided by the present invention, it is not necessary to Figure 1 The steps are executed in the order shown. The specific execution order of the steps can be determined according to needs, and the present invention does not limit this.

[0108] The above is a video reconstruction method based on a video variational autoencoder provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding video reconstruction device based on a video variational autoencoder, such as Figure 12 shown.

[0109] Figure 12 Schematic diagram of a video reconstruction device based on a video variational autoencoder provided by the present invention, the device 1200 includes:

[0110] Construction module 1201 is used to introduce a time domain residual module on the basis of the original image decoder to form a pseudo 3D residual module, and add a unidirectional loop module after the pseudo 3D residual module to form a video decoding module; the original image encoder and the video decoding module form a video variational autoencoder network; the video variational autoencoder network is trained to obtain a video variational autoencoder.

[0111] The encoding module 1202 is used to input the video to be reconstructed into the original image encoder of the video variational autoencoder to obtain the encoding features of the video to be reconstructed in the low-dimensional latent space;

[0112] The decoding module 1203 is used to input the coding features into the video decoding module of the video variational autoencoder, use the time domain residual module in the pseudo three-dimensional residual module to strengthen the time domain local feature interaction between consecutive video frames in the coding features, and in the one-way loop module, transfer the time domain long-distance features of the video frames output by the pseudo three-dimensional residual module through the hidden state including the historical video frame features to obtain the reconstructed video.

[0113] For the specific definition of the video reconstruction device based on the video variational autoencoder, please refer to the definition of the video reconstruction method based on the video variational autoencoder above, which will not be repeated here. The various modules in the above-mentioned video reconstruction device based on the video variational autoencoder can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0114] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The video reconstruction method based on video variational autoencoder is provided.

[0115] The present invention also provides Figure 13 The structural diagram of the computer equipment shown in FIG. Figure 13 As shown in the figure, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The video reconstruction method based on video variational autoencoder is provided.

[0116] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0117] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A video reconstruction method based on video variational autoencoder, characterized in that: include: A one-dimensional time-domain residual module is introduced on the basis of the original image decoder to form a pseudo three-dimensional residual module, and a one-way loop module is added after the pseudo three-dimensional residual module to form a video decoding module; A video variational autoencoder network is formed by the original image encoder and the video decoding module; Training the video variational autoencoder network to obtain a video variational autoencoder; Inputting the video to be reconstructed into the original image encoder of the video variational autoencoder to obtain the encoding features of the video to be reconstructed in the low-dimensional latent space; The encoding features are input into the video decoding module of the video variational autoencoder, and the one-dimensional time domain residual module in the pseudo three-dimensional residual module is used to enhance the interaction of time domain local features between consecutive video frames in the encoding features. In the one-way loop module, the video frames output by the pseudo three-dimensional residual module are subjected to time domain long-distance feature transfer through the hidden state including the historical video frame features to obtain a reconstructed video.

2. The method according to claim 1, characterized in that The video decoding module includes multiple groups of pseudo three-dimensional residual modules and one-way loop modules; each video frame of the video to be reconstructed corresponds to a group of pseudo three-dimensional residual modules and one-way loop modules.

3. The method according to claim 2, characterized in that The pseudo three-dimensional residual module includes a two-dimensional spatial residual module and a one-dimensional temporal residual module; and using the one-dimensional temporal residual module in the pseudo three-dimensional residual module to enhance the temporal local feature interaction between consecutive video frames in the coding feature includes: For any video frame in the coding feature, the video frame and the frames before and after the video frame are input into the corresponding pseudo 3D residual module, and the visual features of the video frame in the spatial domain are obtained through the 2D spatial residual module; The visual features in the spatial domain are input into the one-dimensional temporal residual module to obtain temporal local interaction features after the video frame interacts with adjacent frames in the temporal domain.

4. The method according to claim 3, characterized in that The one-dimensional time domain residual module includes a plurality of time domain operation layers with skip-connected branches; the visual features in the spatial domain are input into the one-dimensional time domain residual module to obtain the time domain local interaction features after the video frame interacts with adjacent frames, including: Inputting the spatial visual features into the multiple temporal operation layers in sequence, and obtaining temporal local correlation features by strengthening temporal local feature correlation between consecutive video frames; The visual features in the spatial domain are fused with the local correlation features in the temporal domain through skip connection branches to obtain the local interaction features in the temporal domain.

5. The method according to claim 4, characterized in that The channel process of the time domain operation layer includes: group normalization, Swish activation, random inactivation layer and one-dimensional convolution in the time domain direction.

6. The method according to claim 2, characterized in that The input end of each unidirectional cycle module includes two parts, one part is connected to the output end of the corresponding pseudo 3D residual module in the depth direction of the model, and the other part is connected to the hidden state output end of the unidirectional cycle module corresponding to the previous video frame in the time domain direction; the unidirectional cycle module includes a window mutual attention feature alignment unit, a convolution unit, a channel splitting unit and a jump connection unit. In the unidirectional cycle module, the video frame output by the pseudo 3D residual module is subjected to time domain long-distance feature transfer through the hidden state including the historical video frame features to obtain a reconstructed video, including: For any video frame, the temporal local interaction feature of the video frame and the hidden state output by the one-way loop module corresponding to the previous video frame are input into the window mutual attention feature alignment unit to obtain an aligned feature; Performing three-channel splicing of the video temporal local interaction feature, the hidden state, and the alignment feature to obtain a spliced ​​feature, and fusing and transforming the spliced ​​feature through a convolution unit to obtain a fused feature; The fusion feature is divided along the feature channel dimension by a channel segmentation unit to obtain a hidden state residual part and a current frame feature residual part; Performing feature addition on the hidden state residual part and the hidden state output by the unidirectional cycle module corresponding to the previous video frame through the jump connection unit to obtain the hidden state output by the unidirectional cycle unit; Adding the feature residual of the current frame and the time-domain local interaction feature of the video frame through the jump connection unit to obtain the decoding feature output by the unidirectional cycle unit; The reconstructed video is obtained according to the decoding features output by each unidirectional cyclic unit.

7. The method according to claim 6, characterized in that The window mutual attention feature alignment unit is established based on the shifted window mutual attention mechanism and is composed of two layers of window mutual attention modules stacked together. The temporal local interaction features of the video frame and the hidden state output by the unidirectional loop module corresponding to the previous video frame are input into the mutual attention feature alignment unit to obtain the alignment features, including: Normalizing the temporal local interaction features of the video frame and the hidden state output by the unidirectional loop module corresponding to the previous video frame, respectively, and performing window division on the normalized temporal local interaction features and hidden state; The temporal local interaction features and hidden states after window division are input into the first-layer window mutual attention module. Under each divided window, the temporal local interaction features of the video frame are used as query features, and the hidden state output by the one-way loop module corresponding to the previous video frame is used as the key and value of mutual attention. Multi-head mutual attention calculation is performed through the introduced sinusoidal position encoding mechanism. The result of the multi-head mutual attention calculation is fused with the temporal local interaction features of the video frame to obtain the alignment features of the first stage. The alignment features of the first stage and the windows of the hidden state are shifted, and the alignment features and hidden states after the window shift are input into the second-layer window mutual attention module. The second-layer window mutual attention module performs multi-head mutual attention calculation on the alignment features after the window shift and the hidden state, and the results of the multi-head mutual attention calculation are fused with the alignment features of the first stage to obtain the alignment features output by the window mutual attention feature alignment unit.

8. The method according to claim 2, characterized in that Each group of pseudo 3D residual modules and unidirectional loop modules are connected via a dual-branch upsampling module, which is obtained by expanding the interpolation upsampling module after the spatial residual module of the original image decoder; the dual-branch upsampling module includes two branches, one branch includes nearest neighbor difference and 2D convolution, and the other branch includes 2D convolution and pixel rearrangement. The method further includes: For any video frame, the temporal local interaction features output by the pseudo 3D residual module are sequentially subjected to nearest neighbor difference and 2D convolution to obtain a first up-sampled feature; The time-domain local interaction features output by the pseudo 3D residual module are sequentially subjected to 2D convolution and pixel rearrangement to obtain a second up-sampled feature; The first up-sampled features and the second up-sampled features are fused, and the fused features are input into the unidirectional loop module.

9. The method according to claim 1, characterized in that The video variational autoencoder includes multiple video decoding modules; the multiple video decoding modules are arranged in series, and the input of the next video decoding module is the output of the previous video decoding module.

10. The method according to claim 1, characterized in that The step of training the video variational autoencoder network to obtain a video variational autoencoder includes: A consistent initialization strategy is used to initialize the parameters of the video variational autoencoder network while maintaining the spatial domain pre-training knowledge to form a video reconstruction model; The video reconstruction model is fine-tuned in the time domain layer on the training video data to obtain the video variational autoencoder.

Images