Method for compressing video super-resolution based on QP adaptive compression prior

Abstract

The application discloses a kind of QP adaptive compression video super-resolution methods based on compression prior, mainly include the following steps: using feature extraction block extracts the feature of compressed video and its compression prior;QP modulation module is constructed, and the feature is recalibrated using frame level QP;Adaptive multi-scale prior information fusion module is constructed to constitute prior fusion module, make full use of the encoding prior in video stream and the feature of video;Using bidirectional propagation alignment module to extract space-time feature and align the feature after fusion;Adaptive enhancement modulation module is constructed to constitute reconstruction module, and the feature is adaptively enhanced;Feature extraction block, prior fusion module, bidirectional propagation alignment module and reconstruction module jointly constitute QP adaptive network, and the network can realize super-resolution to video compressed in different QP.The QP adaptive compression video super-resolution method based on compression prior of the application can obtain better subjective and objective effects.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of compressed video post-processing, in particular to a QP adaptive compressed video super-resolution method based on compressed prior, and belongs to the field of image / video processing. BACKGROUND

[0002] Due to the limitation of network bandwidth and storage space, most videos need to be compressed and encoded before being stored or transmitted. However, under the condition of limited bandwidth, video compression sacrifices video quality for higher compression ratio, which seriously affects the quality of experience of the video. Therefore, a more effective framework is needed to reduce the bit rate and maintain high video quality. Benefiting from the development of video super-resolution technology, the video coding framework based on downsampling (downsampling-encoding-decoding-super-resolution) has shown great potential and attracted widespread interest, and the focus is on the super-resolution reconstruction method of compressed video.

[0003] Although video super-resolution has achieved exciting research results, there are two serious challenges in directly applying existing super-resolution methods to the downsampling-based video coding scheme. The first challenge is that the compressed low-resolution video has serious compression artifacts, and most existing research works only consider bicubic downsampling as the degradation factor, which cannot directly perform effective super-resolution reconstruction on the compressed low-resolution video. The second challenge is that different quantization parameter compressed videos have different degrees of compression artifacts, making it difficult to effectively reconstruct high-resolution videos with the same model. Taking HEVC as an example, the QP value range is 0-51. If the same network model is used for any QP compressed video, the performance will be significantly affected, and training a model for each QP separately is too expensive and not practical. SUMMARY

[0004] The purpose of the present application is to propose an effective adaptive compressed video super-resolution method for compressed video.

[0005] The QP adaptive compressed video super-resolution method based on compressed prior proposed by the present application includes the following steps in the implementation process:

[0006] (1) Multi-scale feature extraction: Specifically, a multi-scale convolutional feature extraction block is used to extract multi-scale shallow features of compressed low-resolution video frames and their corresponding compressed prior information (prediction signal, block diagram, residual);

[0007] (2) Constructing a QP modulation module: Specifically, the module takes the frame-level QP as input, generates a modulation map through a 1x1 convolution layer and a Leakly ReLU layer to form a modulation unit, recalibrates the input features in the channel using the modulation map, and outputs the modulated features;

[0008] (3) Constructing a priori fusion module: specifically, an adaptive multi-scale priori information fusion module is constructed on the basis of the multi-scale convolution and QP modulation module, a plurality of multi-scale priori information fusion modules are stacked to constitute a priori information fusion module, the video spatial multi-scale features and the multi-scale features of the corresponding priori information are fused, and the fused features are outputted;

[0009] (4) Bidirectional propagation alignment: specifically, the fused spatial features are fused in the time domain by using an existing bidirectional propagation alignment module, and the features are aligned, and the aligned features are outputted;

[0010] (5) Constructing a reconstruction module: specifically, an adaptive enhancement modulation module is constructed on the basis of the QP modulation module, and the adaptive enhancement modulation modules are stacked to constitute a reconstruction module, which adaptively enhances the features under the guidance of the frame-level QP;

[0011] (6) Combining the feature extraction block, the multi-scale priori fusion module, the bidirectional propagation alignment module and the reconstruction module into a final QP adaptive network;

[0012] (7) Training the network in step (6) by using training data;

[0013] (8) When testing, taking a low-resolution compressed video, corresponding compressed priori (prediction signal, block diagram, residual) and frame-level QP as the input of the network, and outputting a final super-resolution video. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is an overview of the QP adaptive compressed video super-resolution method based on compressed priori of the application: wherein (a) is a structural block diagram of the QP adaptive network, (b) is a feature extraction block in (a), (c) is a feature fusion block in (a), and (d) is a structural diagram of the multi-scale QP modulation module in (a).

[0015] Figure 2 is a structural diagram of the QP modulation module of the application.

[0016] Figure 3 is a structural diagram of the adaptive multi-scale priori information fusion module of the application.

[0017] Figure 4 is a structural diagram of the adaptive enhancement modulation module of the application.

[0018] Figure 5Fig. 1 is a subjective visual effect comparison chart of the present application and six methods: (a) is the original high-resolution video frame, (b) and (c) are the image block selected for display on the original high-resolution frame and the corresponding decoded low-resolution image block, respectively, (d), (e), (f), (g), (h) and (i) are the subjective visual effect comparison charts of the present application after super-resolution, method 1, method 5, method 6, method 7, method 8, respectively. DETAILED DESCRIPTION

[0019] The present application will be further described below in conjunction with the accompanying drawings:

[0020] The implementation process of the QP adaptive compression video super-resolution method based on compressed prior includes the following steps:

[0021] (1) Multi-scale feature extraction: Specifically, a feature extraction block composed of multi-scale convolution is used to extract multi-scale shallow features of compressed low-resolution video frames and their corresponding compressed prior information (prediction signal, block diagram, residual);

[0022] (2) Constructing a QP modulation module: Specifically, this module takes frame-level QP as input, generates a modulation map through a 1x1 convolution layer and a Leakly ReLU layer to form a modulation unit, recalibrates the input features in the channel using the modulation map, and outputs the modulated features;

[0023] (3) Constructing a prior fusion module: Specifically, an adaptive multi-scale prior information fusion module is constructed based on multi-scale convolution and QP modulation module, multiple multi-scale prior information fusion modules are stacked and a simple feature fusion block is used to form the prior fusion module, which fuses the spatial multi-scale features of the video and the multi-scale features of the corresponding prior information, and outputs the fused features;

[0024] (4) Bi-directional propagation alignment: Specifically, the fused spatial features are fused in the time domain and aligned using the existing bi-directional propagation alignment module, and the aligned features are output;

[0025] (5) Constructing a reconstruction module: Specifically, an adaptive enhancement modulation module is constructed based on the QP modulation module, and the adaptive enhancement modulation module is stacked to form the reconstruction module. Under the guidance of the frame-level QP, the module adaptively enhances the features;

[0026] (6) Combining the feature extraction block, the prior fusion module, the bi-directional propagation alignment module and the reconstruction module into the final QP adaptive network;

[0027] (7) Training the network in step (6) using training data;

[0028] (8) During testing, the low-resolution compressed video and its corresponding compression priors (frame-level QP, prediction signal, block diagram, residual) are used as the input of the network, and the final super-resolution video is output.

[0029] Specifically, the input low-resolution compressed video frame sequence is defined as X. CLR ={x t-r ,…,x t ,…,x t+r The corresponding compressed prior information (predicted signal, block diagram, residual) is defined as X. CP ={(x CP ) t-r ,…,(x CP ) t ,…,(x CP ) t+r The frame level QP is defined as QP = {qp} t-r ,…,qp t ,…qp t+r}, where r represents the radius of the time neighborhood.

[0030] In step (1), using as Figure 1 (b) shows the feature extraction blocks constructed by multi-scale convolutions, which extract low-resolution compressed video X respectively. CLR and corresponding compression prior X CP The multi-scale shallow layer features process can be represented as:

[0031]

[0032]

[0033] Among them, and Indicates from X CLR Extracted high-scale and low-scale features, and f represents the high-scale and low-scale features extracted from the compressed prior of the corresponding video sequence. FEV (·) and f FEP (·) indicates a feature extraction operation.

[0034] The QP modulation module constructed in step (2) is as follows: Figure 2 As shown. Given a feature map As input, a guide map G related to the feature content is generated through a CLC structure and a sigmoid layer, where the CLC structure consists of two 1×1 convolutional layers and one LeaklyReLU layer. Frame-level QPs are then fed into the modulation unit to generate a 1D modulation map. The modulation process can be summarized as follows:

[0035]

[0036] where F M is the feature modulated by the QP modulation module, f CLC (·) denotes the function of the CLC structure, denotes element-wise multiplication. M = f MU (QP), f MU (·) denotes the function of the modulation unit, G = σ(f CLC (F)), and σ denotes the sigmoid function.

[0037] The prior fusion module in step (3) is composed of m adaptive multi-scale prior information fusion modules as shown in FIG. 2B and a feature fusion block as shown in FIG. 2C. Figure 3 The multi-scale shallow features obtained in step 1 and the frame-level QP are input into the prior fusion module to achieve adaptive multi-scale multi-feature fusion, which can be represented as: Figure 1

[0038]

[0039] where F PFM denotes the fused feature, f PFM (·) denotes the function of the prior fusion module.

[0040] The adaptive multi-scale prior information fusion module in step (3) is composed of one multi-scale convolution with C channels, a three-dimensional attention structure, and a multi-scale QP modulation module. Specifically, for the i-th adaptive multi-scale prior information fusion module, the input is the high-scale feature and the low-scale feature generated by the fusion of the previous stage module, and the high-scale feature and the low-scale feature extracted on the compression prior, as well as the frame-level QP. First, the features of different scales are connected in the channel dimension to achieve preliminary fusion, obtaining Then, multi-scale convolution is used to realize communication and aggregation between multi-scale features, which can be represented as:

[0041]

[0042]

[0043] Here ​are the high and low scale features after aggregation, f(·, W) denotes the convolution with parameter W, δ refers to the Leakly ReLU function, upsample(·, s) denotes the nearest neighbor interpolation operation with up-sampling parameter s, pool(·, k) denotes the average pooling operation with k x k kernel and stride k. Then, the three-dimensional attention structure is used to further enhance the representation ability of multi-scale features. We use the energy function E to guide the generation of the weight of three-dimensional attention, and recalibrate the features according to the importance of each position in the feature map. Thereafter, the adaptive fusion is performed with the input high and low features through the skip connection. This process can be summarized as:

[0044]

[0045]

[0046] Here, denote the high and low scale features after adaptive fusion, are the additional 4 learnable parameters initialized to 1, and the energy function E uses the calculation method of Yang, reference “Yang L, Zhang R Y, Li L, et al. Simam: A simple, parameter-free attention module for convolutional neural networks [C] / / International Conference on Machine Learning. PMLR, 2021: 11863-11874.”. is then fed into the multi-scale QP modulation module as shown in Figure 1 (c) together with the frame-level QP. The output of the adaptive multi-scale prior information fusion module is:

[0047]

[0048] Here f MSQPMM (·) denotes the function of the multi-scale QP modulation module. The adaptive multi-scale prior information fusion module not only can adaptively fuse multi-scale features, but also introduces prior information, which improves the representation ability of the model for different QP compressed video features.

[0049] The bidirectional propagation alignment module in step (4) is as shown in Figure 1As shown in (a), its specific structure is consistent with the bidirectional propagation alignment module in Basicvsr++, as referenced in "Chan KCK, Zhou S, Xu X, et al. Basicvsr++: Improving video super-resolution with enhanced propagation and alignment[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022:5972-5981."

[0050] In step (5), the reconstruction module consists of n modules such as... Figure 4 The adaptive enhancement modulation module shown is composed of [a specific module name]. The adaptive enhancement modulation module first employs a wide activation structure to enhance the feature flow; that is, it passes the input features through a 3×3 convolutional layer before the activation layer. The number of channels is expanded from C to s×C, and then a 3×3 convolutional layer is used to adjust the number of activated feature channels to C, resulting in... Subsequently, global average pooling, Leakly ReLU layers, and sigmoid layers are used to generate attention weights for different channels, thereby adaptively recalibrating feature F. WA Adaptive residuals combine the initial features and the recalibrated features using learnable parameters χ and μ to obtain adaptively enhanced features. Finally, a QP modulation module was embedded at the tail, and the adaptively enhanced modulation feature F was obtained by adaptively fusing the modulated features and the initial features using learnable parameters γ and η. EMM :

[0051] F EMM =γ×(f QPMM (C 1×1 (F E )))+η×F in

[0052] Among them, f QPMM (·) represents a function of the QP modulation module, C 1×1 This represents a 1×1 convolution operation.

[0053] The QP adaptive network in step (6) is as follows: Figure 1 As shown in (a), the QP adaptive network is composed of the feature extraction block in step (1), the prior fusion module in step (3), the bidirectional propagation alignment module in step (4), and the reconstruction module in step (5). Through unified training, the loss function of the QP adaptive network is expressed as:

[0054]

[0055] wherein denotes the network reconstructed super-resolution result, y i denotes the original high-resolution video frame, ε is a penalty coefficient, which is set to 1 x 10 -6 , λ is a weight hyper-parameter, which is set to 0.05, L FFT is a loss function based on fast Fourier transform, defined as wherein A(·) denotes the arithmetic square root of the sum of squares of the real part and the imaginary part after Fourier transform.

[0056] During the step (7) training, the hyper-parameters r, C, m, n, s are respectively set to 3, 64, 5, 7, and 2. After down-sampling all the videos in the training set, the test set, and the validation set by 4 times, the compressed low-resolution videos and the compressed priors (frame-level QP, prediction signal, block diagram, residual) are obtained by compressing the videos in LDB and RA modes under 4 different QPs (22, 27, 32, 37) by using the HM16.2 software of HEVC. In the training stage, the video segments with different configurations and different QPs are mixed together to train a general model of ours. The 64 x 64 blocks are randomly cropped from the compressed low-resolution videos, and the 256 x 256 blocks are randomly cropped from the corresponding positions of the original high-resolution videos. Data augmentation is performed by flipping and rotating, and the batch size is set to 4. The ADMM with default parameters is used as the optimizer, the initial learning rate is set to 0.0001, and is reduced to 0.00001 in the training process.

[0057] In order to verify the effectiveness of the present application, comparative experiments are carried out on 10 standard test sequences of the Video Coding Joint Collaborative Group (JCT-VC), whose resolutions are 720p (1280 x 720), 1080p (1920 x 1080), and WQXGA (2560 x 1600). In the experiment, the present application is compared with 8 typical video super-resolution algorithms and 1 compressed video super-resolution algorithm combined with compression priors. The 8 algorithms are:

[0058] Method 1: Method proposed by Jo et al., reference “Jo Y, Oh S W, Kang J, et al. Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 3224-3232.”

[0059] Method 2: Method proposed by Xue et al., reference “Xue T, Chen B, Wu J, et al. Video enhancement with task-oriented flow [J]. International Journal of Computer Vision, 2019, 127: 1106-1125.”

[0060] Method 3: Method proposed by Wang et al., reference “Wang L, Guo Y, Lin Z, et al. Learning for video super-resolution through HR optical flow estimation [C] / / Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part I 14. Springer International Publishing, 2019: 514-529.”

[0061] Method 4: Method proposed by Tian et al., reference “Tian Y, Zhang Y, Fu Y, et al. temporally-deformable alignment network for video super-resolution. In 2020 IEEE [C] / / CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 3357-3366.”

[0062] Method 5: Method proposed by Haris et al., reference “Haris M, Shakhnarovich G, Ukita N. Recurrent back-projection network for video super-resolution [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 3897-3906.”

[0063] Method 6: Method proposed by Wang et al., reference “Wang X, Chan K C K, Yu K, et al. Edvr: Video restoration with enhanced deformable convolutional networks [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition workshops. 2019: 0-0.”

[0064] Method 7: Method proposed by Chan et al., reference “Chan K C K, Wang X, Yu K, et al. Basicvsr: The search for essential components in video super-resolution and beyond [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 4947-4956.”

[0065] Method 8: Method proposed by Chan et al., reference “Chan K C K, Zhou S, Xu X, et al. Basicvsr++: Improving video super-resolution with enhanced propagation and alignment [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 5972-5981.”

[0066] Method 9: Method proposed by Chen et al., reference “Chen P, Yang W, Wang M, et al. Compressed domain deep video super-resolution[J]. IEEE Transactions on Image Processing, 2021, 30: 7156-7169.”

[0067] The content of the comparative experiment is as follows:

[0068] Methods 1, 2, 3, 4, 5, 6, 7, 8, and 9 are retrained with the same training set, and compressed video super-resolution is performed on the test set, and compared with the present method. It is worth noting that methods 1-9 need to train specific models for different encoding configurations and QPs, while the present method only needs to train one model. In this experiment, low-resolution compressed videos under LDB mode, QP = 22, 27, 32, and 37 are super-resolved. Table I gives the comparison of the average PSNR (Peak Signal to Noise Ratio) and SSIM (Structure Similarity Index) of the reconstruction results of each method on the test set. The larger the PSNR (dB) and SSIM values, the better the quality of the reconstructed video. The best results obtained from all experiments are marked in bold. In addition, “BasketballDrive (LDB, QP = 27)”, “BQTerrace (RA, QP = 32)”, “Johnny (RA, QP = 27)”, “KristenAndSara (LDB, QP = 22)”, and “PeopleOnStreet (RA, QP = 27)” are selected for visual comparison, as shown in Figure 5 Figure 5 (a) represents the original high-resolution video frame, Figure 5 (b), Figure 5 (c) are the selected image blocks for display on the original high-resolution frame and the corresponding decoded low-resolution image blocks, respectively. Figure 5 (d), Figure 5 (e), Figure 5 (f), Figure 5 (g), Figure 5 (h) and ​ (i) are the subjective visual effect comparison charts of methods 1, 5, 6, 7, 8, and the present invention after super-resolution, respectively.

[0069] Table I

[0070] ​

[0071] From the experimental results, it can be seen that:

[0072] Methods 1-8 are video super-resolution algorithms, and good results are obtained by retraining using compressed videos, method 9 uses coding priori, and compared with methods 1-7, it has good improvement, and the application fully utilizes the coding priori, and only needs to deploy one model to achieve better objective indicators than methods 1-9 at different QPs. In addition, compared with methods 1, 5, 6, 7, 8, the reconstruction results of the QP adaptive compressed video super-resolution method based on the compression priori proposed in the application are visually pleasing, and the texture structure is clearer.

[0073] In summary, the video reconstructed by the application has obvious advantages in subjective visual effect and objective evaluation indicators. Therefore, the application is an effective compressed video super-resolution method.

Claims

1. A method for QP adaptive compression video super-resolution based on compression priors, characterized in that The implementation process comprises the following steps: Step one: multi-scale feature extraction: specifically, the multi-scale convolutional feature extraction block is used to distinguish and extract the multi-scale shallow features of the compressed low-resolution video frames and the corresponding compressed prior information, which includes prediction signals, block diagrams and residuals; Step two: build a QP modulation module: specifically, this module takes frame-level QP as input, generates a modulation map through a 1x1 convolution layer and a LeaklyReLU layer, recalibrates the input features in the channel using the modulation map, and outputs the modulated features; Step three: build a prior fusion module: specifically, an adaptive multi-scale prior information fusion module is built based on the multi-scale convolution and QP modulation module, multiple multi-scale prior information fusion modules are stacked to form the prior information fusion module, and the video spatial multi-scale features and the corresponding multi-scale features of the prior information are fused to output the fused features; Step four: bidirectional propagation alignment: specifically, the existing bidirectional propagation alignment module is used to fuse the fused spatial features in the time domain and align the features, and output the aligned features; Step five: build a reconstruction module: specifically, an adaptive enhancement modulation module is built based on the QP modulation module, and the adaptive enhancement modulation module is stacked to form the reconstruction module. Under the guidance of the frame-level QP, the module adaptively enhances the features; Step six: combine the feature extraction block, the prior fusion module, the bidirectional propagation alignment module and the reconstruction module into the final QP adaptive network; Step seven: train the network in step six using training data; Step eight: when testing, take the low-resolution compressed video and its corresponding compressed prior as the input of the network, and output the final super-resolution video. The compressed prior in this part includes frame-level QP, prediction signal, block diagram and residual. Specifically, for the QP modulation module in step two, given the feature map As input, the guidance map G related to the feature content is generated by a CLC structure and a sigmoid layer, where the CLC structure is composed of 2 1x1 convolution layers and 1 LeaklyReLU layer, and the frame-level QP is fed into the modulation unit to generate a 1D modulation map The modulation process is: where F M is the feature modulated by the QP modulation module, f CLC (·) denotes a function of the CLC structure, denotes element-wise multiplication, M = f MU (QP), f MU (·) denotes a function of the modulation unit, G = σ(f CLC (F)), σ denotes a sigmoid function; The prior fusion module in step three consists of m adaptive multi-scale prior information fusion modules and feature fusion blocks. For the i-th adaptive multi-scale prior information fusion module, the input is the high-scale feature generated by the fusion of the modules in the previous stage. and low-scale features and high-scale features extracted from compressed priors and low-scale features And frame-level QP, firstly, features of different scales are concatenated along the channel dimension to achieve preliminary fusion, resulting in Then, multi-scale convolution is used to achieve communication and aggregation between multi-scale features. The process is as follows: Here are the high and low scale features after aggregation, f(·, W) denotes the convolution with parameters W, δ refers to the Leakly ReLU function, upsample(·, s) denotes the nearest neighbor interpolation operation with up-sampling parameter s, pool(·, k) denotes the average pooling operation with k x k kernel and stride k; then, the three-dimensional attention structure is used to further enhance the representation ability of multi-scale features, and the energy function E is used to guide the generation of the weight of three-dimensional attention, and the features are adaptively calibrated according to the importance of each position in the feature map, and then the adaptive fusion is performed with the input high and low features through the skip connection, and the process is as follows: Herein respectively represent the high and low scale features after adaptive fusion, are additional 4 learnable parameters initialized as 1, and then fed into a multi-scale QP modulation module together with the frame-level QP, the multi-scale QP modulation module includes two QP modulation modules, the inputs of the module are two shallow features of different scales and the frame-level QP, the two QP modulation modules share the same QP value, and the two QP modulation modules process the shallow features of different scales respectively, and finally output the modulated shallow features of different scales; the output of the final adaptive multi-scale prior information fusion module is: Here f MSQPMM (·) denotes a function of the multi-scale QP modulation module.

2. The method of claim 1, wherein The reconstruction module in step five enhances deep features. Specifically, the reconstruction module consists of n adaptive enhancement and modulation modules. The adaptive enhancement and modulation module first uses a wide activation structure to enhance the feature flow, that is, it passes the input features through a 3×3 convolutional layer before the activation layer. The number of channels is expanded from C to s×C, and then a 3×3 convolutional layer is used to adjust the number of activated feature channels to C, resulting in... Subsequently, global average pooling, Leakly ReLU layers, and sigmoid layers are used to generate attention weights for different channels, thereby adaptively recalibrating feature F. WA ; The adaptive residual combines the initial feature and the recalibrated feature through the learnable parameters χ, μ to obtain the adaptive enhanced feature Finally, the QP modulation module is embedded in the tail, and the modulated feature and the initial feature are adaptively fused through the learnable parameters γ, η to obtain the adaptive enhanced modulation feature F EMM : F EMM = γ x (f QPMM (C 1×1 (F E )) + η x F in where f QPMM (·) denotes a function of the QP modulation module, C 1×1 denotes a 1 x 1 convolution operation.

Images