Real scene video degradation method based on multi-frame space-time fusion

Abstract

The invention discloses a real scene video degradation method based on multi-frame space-time fusion, belongs to the field of computer science, and aims to solve the problem of insufficient scale of an existing real scene data set. A video super-resolution task in a real scene is used for improving the resolution and definition of a low-resolution video which is complex and unknown in degradation and is in the real scene, and at present, some conventional methods adopt a mode of combining bicubic downsampling with Gaussian blur to simulate a video degradation process. However, the too simple degradation methods often result in easy overfitting of the model and lack of generalization. In addition, the number and scale of the existing data set with real scene degradation are limited, and the performance of the model is also limited. According to the method, training is carried out by utilizing the existing real scene data set, complex degradation characteristics in a network learning real scene are designed, and the characteristics are migrated and applied to other conventional data sets, so that the limitation of the data set on the aspects of scale and too simple degradation is solved.

Description

Technical Field

[0001] The present invention belongs to video image processing technology and is mainly applied to the field of video super-resolution. Specifically, it relates to a real scene video degradation method based on multi-frame spatiotemporal fusion. Background Art

[0002] In recent years, with the popularization and performance breakthroughs of network equipment and the widespread application of 5G technology in mobile communications, people have higher quality requirements for browsing images / videos on the Internet. When transmitting images / videos on the Internet, media resources need to be compressed, and existing media compression coding technologies and network transmission will cause varying degrees of quality degradation. Super-resolution technology (SR) has been a research hotspot in the field of computer vision and image processing in recent years. The main goal of this task is to convert low-resolution images / videos into high-resolution images / videos. Existing video degradation methods mainly perform bilinear downsampling on the video frame by frame and generate low-resolution images through Gaussian blurring with different variance σ values. Such a simple processing method makes the trained model tend to overfit. It only learns a simple inverse process and cannot fully simulate the data loss degradation caused by video recording, encoding compression, and network transmission. Therefore, the trained model has weak reconstruction ability for complex input videos and lacks generalization for real-world scenes. To train a high-performance and high-generalization video super-resolution model, a HR-LR video dataset that can represent real-world degradation is required. However, there are few datasets of this type and their scale is limited. Therefore, this paper focuses on how to make full use of existing video datasets to obtain high-quality low-resolution media data with complex degradation effects. Inspired by VisionTransformer, a real-scene video degradation network is proposed to solve the problem of insufficient real-scene training data. Summary of the invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a real scene video degradation method based on multi-frame spatiotemporal fusion. The network includes a video frame encoding module, a spatial self-operation module, a spatiotemporal cross-operation module and a video frame decoding module. Among them, the video frame encoding and decoding module is composed of a series of residual connections and convolution deconvolutions. The structure of the spatial self-operation module refers to the Vision Transformer architecture. At the same time, a spatiotemporal cross-operation module is added to improve the processing capability of temporal information. The degradation network proposed in this patent fully learns the degradation effects of existing real scene videos, and solves the shortcomings of the current real scene video datasets in size by applying the network to traditional video super-resolution datasets, thereby improving the performance and generalization of video super-resolution methods in real scenes.

[0004] Technical solution: The real scene video degradation method based on multi-frame spatiotemporal fusion described in the present invention includes a video frame encoding module, a timing information extraction module, a spatiotemporal fusion module and a video frame decoding module.

[0005] The degradation of real-scene videos is affected by factors such as motion blur, defocus, signal-dependent noise, anisotropic blur, and video compression algorithms during shooting, showing a nonlinear relationship. In ordinary video super-resolution tasks, bicubic downsampling combined with Gaussian blur is usually used to simulate degradation, so that the model is prone to overfitting and performs worse than expected in real scenes. This patent focuses on proposing a model to implicitly fit this complex degradation distribution. Thanks to the long-distance dependency capture and multi-head attention mechanism of the Transformer architecture to enable the model to learn different degradation features in different subspaces, this patent uses the Encoder structure to construct a spatial self-operation module to capture the dependencies between various positions in the video frame, and uses the Decoder structure to construct a spatiotemporal fusion module to generate a target potential representation sequence, and ensure that the generated sequence has the fusion information of the context and the current time step potential representation.

[0006] In order to further improve the performance of the model and further fit the degradation effect of the video data in network transmission, the present invention adds a network push-pull stream system to further preprocess the video data set. The network push-pull stream consists of three modules: a video push stream end, a streaming media server end, and a video pull stream end. The video push stream end complexly reads the video data and encodes and pushes it to the streaming media server. The streaming media server is responsible for receiving the pushed video stream and publishing it. The pull stream end pulls the specified video stream from the streaming media server according to the stream address provided by the push stream end and decodes and saves it. After this series of operations, a complex degraded video transmitted through network coding is obtained on the basis of the complex degraded video.

[0007] For the video frame encoding module, this module mainly uses a series of convolutional layers to extract the potential representation of the video frame in high dimensions. First, a set of dimensions (B, H f ,W f ,3) is fed into the video encoding module, where B is the batch size, W f ,H f Respectively represent the width and height of the video frame, 3 represents the RGB channel of the video frame, and the video encoding module consists of three layers of convolution with a kernel size of 5 and a stride of 2. A Leaky ReLu activation function layer is added after each convolution layer. Through the processing of this module, each video frame is extracted into a dimension of Processing the latent representation in a high-dimensional latent space can effectively reduce the amount of computation and process more temporal and spatial information in the data fusion stage.

[0008] In the timing information extraction module, the input of this module is l t-2 ,l t-1 represents the latent representation extracted by the video frame encoding module, l t-2 ,l t-1 is the potential representation of the video frame in the previous two steps. This module has two parts, which process the potential representation of the previous time step l t-2 ,l t-1 and the latent representation l that is merged across the two time steps according to the spatial dimension joint Each block consists of a different number of TransformerBlocks, and each block consists of two multi-head attention layers. In the network of the present invention, the multi-head attention layers are divided into two types according to the calculation of the QKV matrix: spatial self-operation and spatiotemporal fusion operation. The spatial self-operation layer calculates the QKV matrix in the current time step itself, while the spatiotemporal cross-operation layer first calculates the joint potential representation l of the previous time step. pre The KV matrix is ​​calculated in , the Q matrix is ​​calculated in the potential representation of the current time step, and finally the temporal spatial information of the potential representation is fused through QKV calculation.

[0009] The Transformer Block module mentioned above can be divided into the spatial self-operation module (SSB) and the spatiotemporal data fusion module (STFB) according to the difference between the encoder and the decoder. The difference between the two modules lies in the operation of the QKV matrix and the multi-head attention algorithm used. Figure 3 As shown in the figure, a) is the spatial self-operation module, which first divides the input potential representation into patches and linearly projects it. After the layer normalization operation, it is divided in the spatial dimension according to the number of attention heads, and then the QKV matrix is ​​calculated to complete the multi-head attention operation; b) is based on the potential representation l provided by the temporal information extraction module pre Calculate the KV matrix according to l cur Calculate the Q matrix, and then use the masked self-attention algorithm to calculate the degraded representation in different subspaces. The detailed calculation process is as follows. Given a patch size w p , number of attention heads num h , the module input dimension is (B, W l , H l , 192), first transform the latent representation according to w p Divided into patch sequences with dimensions A tensor of, where B′=B×N h ×N w , N h With N w The number of patches divided in height and width respectively; then add a dimension to the tensor The position codes are then sent to W q , W k , W v The corresponding QKV matrix is ​​calculated in the fully connected layer and the number of attention heads num h The QKV matrix is ​​split and dimensionally adjusted according to the channel dimension to obtain a dimension of The tensor is used for multi-head self-attention operation. As shown in formula (3), the QKV matrix divided according to the channel dimension is used for attention operation to obtain the output feature F out Finally, the dimension is adjusted and the channels are merged again to make F out The same dimension as the input is used for the next layer of operations. Each Transformer Block superimposes two layers of the above operations, forming different processing modules according to different processing methods for different types of potential representations.

[0010] In the spatial self-operation module, in addition to processing l t-2 , l t-1 The potential representation of In addition, there is an additional spatial self-operation module, which first merges the two outputs according to the spatial dimension to obtain a dimension of The potential representation of should be subjected to the above multi-head self-attention operation to obtain l pre As input to the spatiotemporal fusion module.

[0011] In the spatiotemporal fusion operation module, the potential representations l at different time steps are first t-2 , l t-1 The dimension obtained by operation is Tensor, then l t-2 , l t-1 The tensor calculated is The dimensions are merged and the spatial self-operation is performed again to obtain the dimension The fusion potential representation l pre . Then, in the multi-head self-attention operation of the next layer, l is used pre Calculate the KV matrix of the same dimension, using l t Compute the query matrix Q. The spatiotemporal data is fully integrated in the mask attention calculation and fed into the video frame decoder for reconstruction.

[0012] For the video frame decoder module, the input is the potential representation l of the fused spatiotemporal data calculated by the spatial self-operation module out , whose dimensions are First, the latent representation with the same dimension as the output of the video frame encoding module is obtained by patch merging. Then, the latent representation is first passed through four residual connection blocks and then through a deconvolution to reduce the number of channels and increase the resolution. Then, the latent representation is converted into an image matrix with the dimension B',H,W,3) through two residual connections and a deconvolution. During the training process, the mean square error MSE is selected as the loss function, and the image matrix is ​​compared with the low-resolution video frame data at the pixel level. Then, the gradient is calculated and the model parameters are updated through the back-propagation algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart of the present invention;

[0014] Figure 2 This is a model architecture diagram in the present invention;

[0015] Figure 3 This is a structural diagram of the spatial self-computing layer SSB and the spatiotemporal fusion layer STFB in the present invention; DETAILED DESCRIPTION

[0016] The present invention discloses a real scene video degradation method based on multi-frame spatiotemporal fusion, which is further described below in conjunction with the accompanying drawings and embodiments.

[0017] The present invention discloses a real scene video degradation method based on multi-frame spatiotemporal fusion, comprising a push-pull stream preprocessing module, a video frame encoding module, a spatial self-operation module, a timing information extraction module, a spatiotemporal fusion operation module and a video frame decoding module.

[0018] First, the degenerate network includes the following steps:

[0019] Step (1): In the training phase, the LR data of the video data is preprocessed through a network coding transmission module;

[0020] Step (2): In the video frame encoder stage, the HR-LR frame pair of the network coded transmission degraded video frame obtained in step (1) is extracted using a four-layer convolutional module. Each convolutional module consists of a convolutional unit and a Leaky ReLu unit. Each layer of the convolutional unit reduces the spatial dimension by two times and then expands the channel dimension. The module finally obtains a dimension of The latent space representation of

[0021] Step (3): Based on the latent representation l extracted in step (2) t-1 ,l t-2 ,l t , represents the potential representation extracted in the t-2, t-1, t time steps, where l t-2 ,lt-1 First, it is sent to the timing information extraction module, l t Send it to the time-space cross-fusion module to wait for further calculation;

[0022] Step (4): In both the temporal information extraction module and the spatiotemporal cross fusion module, the spatial self-operation module processes the input potential representation. In the spatial self-operation module, the structure is as follows Figure 3 (a), referring to the image processing in VIT, an 8×8 convolution is used to extract, embed and positionally encode the latent representation, and then a multi-head self-attention operation is performed to obtain the output for the operation of each module. Assuming the patch size is p=8, we use a convolutional layer with a convolution kernel size of p and an output channel number of c to extract the latent representation l; then a fully connected layer is used to linearly project the extracted patch and positionally encode it to obtain Embedded Patches, whose dimension is (B×n h ×n w , p 2 , C), where B represents the batch size, n h , n w Indicates the current l t The number of patches divided in height and width respectively, C is the number of output channels of the fully connected layer.

[0023] In the multi-head self-attention operation, the input potential representation is first divided equally according to the number of attention heads in the channel dimension, and then the attention head dimension is swapped with the patch sequence dimension, and then the three different fully connected W q , W k , W v The weights are calculated to obtain the Query matrix Q, Key matrix K and Value matrix V, which are expressed as follows:

[0024] Q=W Q ·X;K=W K ·X;V=W V ·X (1)

[0025] Where B′=B×n h ×n w is the batch size, and C is the number of channels. Then the channel dimension is divided by the number of attention heads H, and the order of the attention head dimension and the patch size dimension is exchanged to convert it into a four-dimensional tensor: Represents the number of embedded channels each attention head has. After obtaining the QKV matrix, the QKV matrix is ​​self-attentioned, and then the attention head dimension is reduced to the channel dimension to obtain the same potential representation as the output dimension. The attention operation expression is as follows:

[0026]

[0027] The tensor Y with the same dimension as the input X is obtained by the operation of formula (3), and the input X is added element by element. Then, the normalization operation is performed through the Layer-Normalization layer. Finally, the attention heads in Y are merged into a tensor with the same dimension as the input to obtain the output. The expression of layer normalization is as follows:

[0028]

[0029] Where μ and σ are the mean and variance of the output of the current layer, respectively, and x i is the input of layer normalization, ∈ is a very small value to prevent zero division errors, and after adjusting the mean and variance, Finally, the layer normalized output y is obtained through two learnable parameters γ and β i Through the above operations, the model output is standardized to reduce internal covariate transfer and avoid the problems of gradient explosion and gradient disappearance.

[0030] In order to process data from two time steps, this patent extracts information from two time steps by stacking spatial self-operation layers in the time series information extraction module for fusion operation with the current time step in the spatiotemporal information fusion module.

[0031] The spatial self-operation layer is calculated through the above process twice. The potential representations of different time steps are calculated in different subspaces through the multi-head self-attention mechanism to obtain the semantic information containing the input potential representation, which is used for encoding and operation in different subsequent steps.

[0032] This patent uses different modules to process different video frames. Assume that l t-2 , l t-1 , l t is the input video frame of the model, l t is the video frame of the current time step, the model extracts l t-2 , l t-1 The time sequence information of the time step is used to t Specifically, for the potential representation of the previous time step, the spatial self-operation layer is first calculated, and then the two tensors are merged in the patch size dimension for the output tensor, and the spatial self-operation is performed again to obtain a dimension of (B′, 2p 2 , C) Latent representation with sufficient spatiotemporal data l pre ; For the potential representation of the current time step, it is first calculated by the spatial self-operation layer of the same structure, and then in the spatiotemporal fusion module, using l pre Calculate K, V matrices, using lt Calculate the Q matrix, where the dimensions of the KV matrix are The dimension of the Q matrix is Finally, in the multi-layer multi-head self-attention operation, the spatiotemporal data is fully integrated according to formula (3) to obtain l out Used by the decoder for reconstruction.

[0033] Step (5): In the time series information extraction module, the calculation results of the t-2 and t-1 time steps are obtained through spatial self-operation. After that, the two potential representations are merged according to the patch size dimension to obtain Then to l joint Perform spatial self-operation to obtain Used for degraded reconstruction in the spatiotemporal cross-fusion module.

[0034] Step (6): In the spatial cross-fusion module, firstly, the latent representation l t After performing the spatial self-operation layer, we get Then, the output l in the module is extracted according to the timing information out Calculate the key value matrix according to formula (2) Using l′ t Compute the query matrix Then, the fusion tensor is calculated according to formula (3) and the attention head dimensions are merged according to the patch size dimension; finally, the output patch sequence is depatch-operated and merged into The latent representation of is used by the frame decoder for degraded reconstruction.

[0035] Step (7): In the video frame decoder module, the module consists of BottleNeck residual blocks and deconvolution. The residual block contains convolutions with kernel sizes of 1×1, 3×3, and 1×1, batch normalization layers, and activation function ReLu layers, which are suitable for data propagation in deeper networks. Deconvolution is used to increase the spatial dimension of the potential representation and reduce the number of channels. For the potential representation input that integrates spatiotemporal information First, it passes through 4 layers of residual blocks and then is sent to the deconvolution with a convolution kernel size of 5×5 to increase the spatial dimension and reduce the number of channels to obtain l′ out ∈ Next, we continue to calculate the residual block and use deconvolution with a convolution kernel size of 1×1 to reduce the channel dimension to 12. Finally, after the last layer of deconvolution operation, we get a low-resolution video frame with a spatial dimension scaled by 4 times and real scene degradation.

[0036] In the model training process, this patent uses mean square error MSE as the loss function, and uses peak signal-to-noise ratio PSNR and structural similarity SSIM as performance comparison benchmarks. The optimizer uses the Adam optimizer, the learning rate is set to 0.001, the training batch size is set to 4, and the training cycle is set to 40. The loss function MSE expression is as follows:

[0037]

[0038] Where n is the number of samples, y i represents the true LR image of the i-th video frame, Represents the i-th reconstructed LR image. The mean square error can effectively reflect the difference between the prediction and the true value at the pixel level.

Claims

1. A real scene video degradation method based on multi-frame spatiotemporal fusion, the multi-frame spatiotemporal fusion method comprising a spatial self-operation module, a temporal information extraction module, a spatiotemporal fusion module, a video frame encoding module, a degradation reconstruction module, and a network coding transmission module; The specific steps of the degradation method are as follows: Step (1): Use the MVSR4X dataset with complex degradation effects in the real world as the training dataset of the degradation model, and use the VideoLQ dataset as the validation dataset; in addition, before model training, input the MVSR4X dataset into the network coding module to obtain video data that has been further degraded by network coding transmission for training; Step (2): The data is read in the form of HR-LR video frame pairs and sent to the video frame encoding module to extract the potential representation of the current video frame for the operation of the next module; Step (3): Based on the latent representation extracted in step (2), first convert the latent representation l calculated for the first two frames into t-1 ,l t-2 The fused latent representation data l with rich spatiotemporal information is obtained. pre For reconstruction; for the potential representation l at the current time step t Perform a single spatial self-operation to obtain the potential representation l of the video frame cur Specifically, the convolutional layer is first used to scan the spatial dimension in the order of raster scanning to extract the patch and perform linear projection and position encoding. The result is first sent to the spatial self-operation module, and then in the spatiotemporal fusion module, the latent representation l of the current time step is used. cur Calculate the Q matrix, according to l pre Calculate the K and V matrices, and finally, after multiple layers of calculation, fully integrate the spatiotemporal data to obtain the latent space representation l out As the input of the degraded reconstruction module. Step (4): In the reconstruction module, the module receives the latent space representation l calculated by the spatiotemporal fusion module out And reconstruct the complex degraded LR video frame data. This module consists of deconvolution and residual blocks. Deconvolution is used to restore the spatial dimension, and the residual block is responsible for controlling the visual effect of the reconstructed video frame. In the model training stage, the final output is compared with the LR video frame in the HR-LR video frame pair, and the mean square error is used as the loss function to train the model.

2. A real scene video degradation method based on multi-frame spatiotemporal fusion according to claim 1, characterized in that: In step (1), the MVSR4X dataset is subjected to network coding transmission to further obtain the degradation effect of video data during network transmission. In terms of building the network transmission module, H.264 is used as the encoding algorithm, RTMP is used as the streaming protocol, SRS server is used as the streaming server, and FFmpeg is used on the client to encode and decode video frames, push and pull streams, and read and save operations.

3. According to the real scene video degradation method based on multi-frame spatiotemporal fusion as described in claim 1, it is characterized in that: In step (2), a four-layer convolutional network is used to extract the latent space representation of the degraded video data transmitted through network coding input in step (1). Each convolutional layer consists of a convolution unit and a LeakyReLu linear correction unit. The convolution kernel size in the convolution module is set to 5×5, the step size is set to 2, and the number of channels is changed from 3 to 192, and the spatial size is reduced to 1 / 16 of the original size. The expression of the Leaky ReLu unit is as follows: Among them, α = 0.01 is used to control the data to tend to 0 while preventing the problem of "dead ReLu" caused by most data being equal to 0.

4. According to the real scene video degradation method based on multi-frame spatiotemporal fusion as described in claim 1, it is characterized in that: In step (3), we first process the latent representation l in the first two time steps. t-2 , l t-1 , which is sent to the temporal information extraction module, and the potential representation of the current frame l t Then it is sent to the spatiotemporal fusion module to wait for the final information extracted in sequence to be fused. Each potential representation will use a sliding window to extract the corresponding patch block, and then merge it according to the spatial dimension to obtain a dimension of A sequence of p Indicates the spatial size of the patch block, d p Indicates the number of channels of the patch block; in the temporal information extraction module, there are two spatial self-operation modules consisting of Transformer encoder branches that perform position encoding, linear projection, and multi-head attention operations on the two potential representations, and then send the two operation results to an additional spatial self-operation module to obtain the final fused potential representation l pre For the potential representation l at the current time step t Then it is input into the backbone network, and first passes through the spatial self-operation layer to perform position encoding, linear projection and multi-head attention operation to obtain the current frame sequence l cur In order to be able to operate with the fused potential representation with temporal information calculated in the previous branch network, the spatiotemporal fusion module pre Calculate the K and V matrices on l cur The query matrix Q is calculated above, and then the QKV matrix is ​​used to fuse the spatiotemporal information in the multi-head attention calculation. In addition, this module stacks multiple layers to ensure full fusion and alignment of information. Inspired by the visual attention of Vision Transformer (VIT), this patent constructs a spatiotemporal information extraction module to extract the timing information of the previous time step; in addition, the spatiotemporal fusion module is further designed with reference to the TransformerDecoder structure to fuse the video frame data of different time steps. Compared with VIT, this module retains the masked multi-head self-attention mechanism operation in the Decoder for the alignment and fusion of the latent space data of the previous and next frames. First, a convolution kernel of size 8×8 and stride 4 is used to perform patch segmentation, linear projection and position encoding on the latent representation; secondly, the latent representation of the video frame in the previous time step is retained and calculated in the temporal information extraction module and fully calculated and fused with the latent representation of the current frame in the spatiotemporal cross fusion module to better meet the needs of complex degraded reconstruction; finally, after multiple layers of iterative fusion, the latent representation rich in spatiotemporal data is obtained for degraded reconstruction in the video frame decoding module.

5. The real scene video degradation method based on multi-frame spatiotemporal fusion according to claim 1, characterized in that: In the video frame decoder stage in step (4), the latent representation output l that fully integrates the spatiotemporal data in step (3) out The video frame is reconstructed into a video frame with real scene degradation through the video frame decoder. The decoder consists of a residual connection module and a deconvolution module. The residual connection module adopts the classic BottleNeck structure in ResNet, which consists of three convolutions with kernel sizes of 1, 3, and 1 and a residual connection. The deconvolution kernel sizes are 5 and 1 respectively. The deconvolution with a kernel size of 5 is used to restore the spatial size of the potential representation to the target size, and the deconvolution with a kernel size of 1 is responsible for mapping the number of channels to image channels to finally obtain the reconstructed LR video frame.

Images