Compression prior-based QP adaptive video super-resolution compression method

Abstract

The invention discloses a QP self-adaptive compressed video super-resolution method based on compression prior. The method mainly comprises the following steps that a feature extraction block is used for extracting features of a compressed video and the compression prior of the compressed video; constructing a QP modulation module, and recalibrating features by using a frame level QP; constructing a self-adaptive multi-scale prior information fusion module to form a prior fusion module, and fully utilizing coding prior and video features in a video stream; a bidirectional propagation alignment module is used to extract spatial and temporal features and align the fused features; constructing an adaptive enhancement modulation module to form a reconstruction module, and adaptively enhancing features; the feature extraction block, the prior fusion module, the bidirectional propagation alignment module and the reconstruction module jointly form a QP adaptive network, and the network can realize super-resolution for different QP compressed videos. According to the QP self-adaptive compressed video super-resolution method based on compression prior, good subjective and objective effects can be obtained.

Description

Technical Field

[0001] The invention relates to a compressed video post-processing technology, and in particular to a QP adaptive compressed video super-resolution method based on compression priori, and belongs to the field of image / video processing. Background Art

[0002] Limited by network bandwidth and storage space, most videos need to be compressed and encoded before being stored or transmitted. However, under extreme bandwidth conditions, video compression sacrifices video quality for a higher compression ratio, which seriously affects the quality of video experience. Therefore, a more effective framework is needed to reduce the bit rate and maintain high video quality. Benefiting from the development of video super-resolution technology, the downsampling-based video coding and decoding framework (downsampling-encoding-decoding-super-resolution) has shown great potential and has attracted widespread interest, with the focus on super-resolution reconstruction methods for compressed videos.

[0003] Although video super-resolution has achieved exciting research results, there are two severe challenges in directly applying existing super-resolution methods to downsampling-based video coding schemes. The first challenge is that the compressed low-resolution video has serious compression artifacts, and most existing research works only use bicubic downsampling as a degradation factor, which cannot directly perform effective super-resolution reconstruction on the compressed low-resolution video. The second challenge is that videos compressed with different quantization parameters have different degrees of compression artifacts, making it difficult to effectively reconstruct high-resolution videos using the same model. Taking HEVC as an example, the QP value range is 0-51. If the compressed video of any QP uses the same network model, the performance will be significantly affected. Training a model for each QP separately is too expensive and not practical. Summary of the invention

[0004] The purpose of the present invention is to propose an effective adaptive compressed video super-resolution method for compressed video.

[0005] The implementation process of the QP adaptive compression video super-resolution method based on compression prior proposed by the present invention includes the following steps:

[0006] (1) Multi-scale feature extraction: Specifically, the feature extraction block composed of multi-scale convolution is used to extract the multi-scale shallow features of the compressed low-resolution video frame and its corresponding compressed prior information (prediction signal, block map, residual);

[0007] (2) Constructing a QP modulation module: Specifically, this module takes the frame-level QP as input, generates a modulation map through a 1×1 convolutional layer and a leaky ReLU layer to form a modulation unit, uses the modulation map to recalibrate the input features on the channel, and outputs the modulated features;

[0008] (3) Constructing a priori fusion module: Specifically, an adaptive multi-scale prior information fusion module is constructed based on the multi-scale convolution and QP modulation modules. Multiple multi-scale prior information fusion modules are stacked to form a prior information fusion module, which fuses the multi-scale features of the video space and the multi-scale features of the corresponding prior information, and outputs the fused features.

[0009] (4) Bidirectional propagation alignment: Specifically, the existing bidirectional propagation alignment module is used to fuse the fused spatial features in the time domain, perform feature alignment, and output the aligned features;

[0010] (5) Constructing a reconstruction module: Specifically, an adaptive enhancement modulation module is first constructed based on the QP modulation module. The adaptive enhancement modulation modules are stacked to form a reconstruction module. This module adaptively enhances features under the guidance of the frame-level QP.

[0011] (6) Combining the feature extraction block, multi-scale prior fusion module, bidirectional propagation alignment module and reconstruction module into the final QP adaptive network;

[0012] (7) training the network in step (6) using the training data;

[0013] (8) During testing, the low-resolution compressed video, its corresponding compression prior (prediction signal, block map, residual) and frame-level QP are used as the input of the network to output the final super-resolution video. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is an overview of the QP adaptive compressed video super-resolution method based on compression prior in the present invention: (a) is a block diagram of the QP adaptive network structure, (b) is the feature extraction block in (a), (c) is the feature fusion block in (a), and (d) is a structural diagram of the multi-scale QP modulation module in (a).

[0015] Figure 2 It is a structural diagram of the QP modulation module of the present invention.

[0016] Figure 3 It is a structural diagram of the adaptive multi-scale prior information fusion module of the present invention.

[0017] Figure 4 It is a structural diagram of the adaptive enhanced modulation module of the present invention.

[0018] Figure 5: is a comparison chart of the subjective visual effects of the present invention and six methods: (a) is the original high-resolution video frame, (b) and (c) are image blocks selected for display on the original high-resolution frame and their corresponding low-resolution image blocks after decoding, (d), (e), (f), (g), (h) and (i) are comparison charts of the subjective visual effects after super-resolution of method 1, method 5, method 6, method 7, method 8 and the present invention, respectively. DETAILED DESCRIPTION

[0019] The present invention will be further described below in conjunction with the accompanying drawings:

[0020] The QP adaptive compression video super-resolution method based on compression prior includes the following steps:

[0021] (1) Multi-scale feature extraction: Specifically, the feature extraction block composed of multi-scale convolution is used to extract the multi-scale shallow features of the compressed low-resolution video frame and its corresponding compressed prior information (prediction signal, block map, residual);

[0022] (2) Constructing a QP modulation module: Specifically, this module takes the frame-level QP as input, generates a modulation map through a 1×1 convolutional layer and a leaky ReLU layer to form a modulation unit, uses the modulation map to recalibrate the input features on the channel, and outputs the modulated features;

[0023] (3) Constructing a priori fusion module: Specifically, an adaptive multi-scale prior information fusion module is constructed based on the multi-scale convolution and QP modulation modules. Multiple multi-scale prior information fusion modules are stacked and a simple feature fusion block constitutes the prior fusion module, which fuses the multi-scale features of the video space and the multi-scale features of the corresponding prior information, and outputs the fused features.

[0024] (4) Bidirectional propagation alignment: Specifically, the existing bidirectional propagation alignment module is used to fuse the fused spatial features in the time domain, perform feature alignment, and output the aligned features;

[0025] (5) Constructing a reconstruction module: Specifically, an adaptive enhancement modulation module is first constructed based on the QP modulation module. The adaptive enhancement modulation modules are stacked to form a reconstruction module. This module adaptively enhances features under the guidance of the frame-level QP.

[0026] (6) Combining the feature extraction block, the prior fusion module, the bidirectional propagation alignment module and the reconstruction module into the final QP adaptive network;

[0027] (7) training the network in step (6) using the training data;

[0028] (8) During testing, the low-resolution compressed video and its corresponding compression prior (frame-level QP, prediction signal, block map, residual) are used as the input of the network, and the final super-resolution video is output.

[0029] Specifically, the input low-resolution compressed video frame sequence is defined as X CLR ={x t-r ,…,x t ,…,x t+r}, and its corresponding compressed prior information (prediction signal, block map, residual) is defined as X CP ={(x CP ) t-r ,…,(x CP ) t ,…,(x CP ) t+r}, the frame level QP is defined as QP = {qp t-r ,…,qp t ,…qp t+r}, where r represents the radius of the time neighborhood.

[0030] In the step (1), using Figure 1 The feature extraction blocks composed of multi-scale convolution shown in (b) extract the low-resolution compressed video X CLR and the corresponding compressed prior X CP The multi-scale shallow features of this process can be expressed as:

[0031]

[0032]

[0033] Among them, and Indicates that from X CLR The extracted high-scale features and low-scale features, and represents the high-scale features and low-scale features extracted from the compressed prior of the corresponding video sequence, f FEV (·) and f FEP (·) indicates feature extraction operation.

[0034] The QP modulation module constructed in step (2) is as follows: Figure 2 As shown. Given a feature map As input, a CLC structure and a sigmoid layer are used to generate a guidance map G related to the feature content. The CLC structure consists of two 1×1 convolutional layers and one LeaklyReLU layer. The frame-level QP is fed into the modulation unit to generate a 1D modulation map The modulation process can be summarized as:

[0035]

[0036] Among them, F M is the characteristic after modulation by the QP modulation module, f CLC (·) represents the function of CLC structure, Represents element-by-element multiplication. M = f MU (QP),f MU (·) represents the function of the modulation unit, G = σ(f CLC (F)),σ represents the sigmoid function.

[0037] The prior fusion module in step (3) consists of m Figure 3 The adaptive multi-scale prior information fusion module shown in Figure 1 (c) shows the feature fusion block structure. The multi-scale shallow features and frame-level QP obtained in step 1 are input into the prior fusion module to achieve adaptive multi-scale multi-feature fusion. The process can be expressed as:

[0038]

[0039] Among them, F PFM represents the fused features, f PFM (·) represents the function of the prior fusion module.

[0040] The adaptive multi-scale prior information fusion module in step (3) is composed of a multi-scale convolution with C channels, a three-dimensional attention structure and a multi-scale QP modulation module. Specifically, for the i-th adaptive multi-scale prior information fusion module, the input is the high-scale feature generated by the fusion of the modules in the previous stage. and low-scale features and high-scale features extracted on the compressed prior and low-scale features As well as the frame-level QP, the features of different scales are first connected in the channel dimension to achieve preliminary fusion, and we get Then, multi-scale convolution is used to realize the communication and aggregation between multi-scale features. The process can be expressed as:

[0041]

[0042]

[0043] here They are respectively the high-scale features and low-scale features after aggregation, f(·,W) represents the convolution with parameter W, δ refers to the Leakly ReLU function, upsample(·,s) represents the nearest neighbor interpolation operation with upsampling parameter s, and pool(·,k) represents the average pooling operation with k×k and step size k. Next, the three-dimensional attention structure is used to further enhance the representation ability of multi-scale features. We use the energy function E to guide the generation of the three-dimensional attention weights, re-adaptively calibrate the features according to the importance of each position in the feature map, and then adaptively fuse them with the high and low features of the input through skip connections. This process can be summarized as:

[0044]

[0045]

[0046] Here Respectively represent the high- and low-scale features after adaptive fusion, There are 4 additional learnable parameters initialized to 1. The energy function E is calculated using Yang's method. Reference "Yang L, Zhang RY, Li L, et al. Simam: A simple, parameter-free attention module for convolutional neural networks [C] / / International Conference on Machine Learning. PMLR, 2021: 11863-11874.". This is then fed together with the frame level QP into Figure 1 (c) The multi-scale QP modulation module shown in Figure 2. The output of the final adaptive multi-scale prior information fusion module is:

[0047]

[0048] Here f MSQPMM (·) represents the function of the multi-scale QP modulation module. The adaptive multi-scale prior information fusion module can not only adaptively fuse multi-scale features, but also introduce prior information to improve the model's ability to represent the features of videos compressed with different QPs.

[0049] The bidirectional propagation alignment module in step (4) is as follows Figure 1As shown in (a), its specific structure is consistent with the bidirectional propagation alignment module in Basicvsr++, reference "Chan KCK, Zhou S, Xu X, et al. Basicvsr++: Improving video super-resolution with enhanced propagation and alignment[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 5972-5981."

[0050] In step (5), the reconstruction module consists of n Figure 4 The adaptive enhancement modulation module is composed of the following components. The adaptive enhancement modulation module first uses a wide activation structure to enhance the feature flow, that is, the input feature is transformed into the feature vector by a 3×3 convolution layer before the activation layer. The channels are expanded from C to s×C, and then a 3×3 convolution layer is used to adjust the number of feature channels after activation to C, and we get Then, global average pooling, Leaky ReLU layer and sigmoid layer are used to generate attention weights of different channels to adaptively recalibrate the feature F. WA The adaptive residual combines the initial features and the recalibrated features through learnable parameters χ, μ to obtain adaptively enhanced features Finally, a QP modulation module is embedded at the tail, and the modulated features and initial features are adaptively fused through learnable parameters γ and η to obtain the adaptive enhanced modulation feature F EMM :

[0051] F EMM =γ×(f QPMM (C 1×1 (F E )))+η×F in

[0052] Among them, f QPMM (·) represents the function of the QP modulation module, C 1×1 Represents a 1×1 convolution operation.

[0053] The QP adaptive network in step (6) is as follows: Figure 1 As shown in (a), it is composed of the feature extraction block in step (1), the prior fusion module in step (3), the bidirectional propagation alignment module in step (4) and the reconstruction module in step (5). The loss function of the QP adaptive network is expressed as follows:

[0054]

[0055] in Represents the super-resolution result of network reconstruction, y i represents the original high-resolution video frame, ε is the penalty coefficient, which is set to 1×10 -6 , λ is the weight hyperparameter, which is set to 0.05 here, L FFT is a loss function based on fast Fourier transform, defined as A(·) represents the arithmetic square root of the sum of the squares of the real part and the imaginary part after Fourier transformation.

[0056] During the training of step (7), the hyperparameters r, C, m, n, s are set to 3, 64, 5, 7, 2 respectively. All videos of the training set, test set and validation set are downsampled by 4 times and compressed by HEVC HM16.2 software in LDB and RA modes at 4 different QPs (22, 27, 32, 37) to obtain compressed low-resolution videos and compressed priors (frame-level QP, prediction signal, block map, residual). In the training phase, video clips with different configurations and different QPs are mixed together to train a general model of ours, with 64×64 blocks randomly cropped from the compressed low-resolution video and 256×256 blocks randomly cropped from the original high-resolution video at the corresponding position, and data augmented by flipping and rotation, and the batch size is set to 4. ADMM with default parameters is used as the optimizer, and the initial learning rate is set to 0.0001, which is reduced to 0.00001 during the training process.

[0057] In order to verify the effectiveness of the present invention, comparative experiments were carried out on 10 standard test sequences of the Joint Collaborative Team on Video Coding (JCT-VC), whose resolutions were 720p (1280×720), 1080p (1920×1080) and WQXGA (2560×1600). In the experiment, the method of the present invention was compared with 8 typical video super-resolution algorithms and 1 compressed video super-resolution algorithm combined with compression prior. The 8 algorithms are:

[0058] Method 1: The method proposed by Jo et al., reference "Jo Y, Oh SW, Kang J, et al. Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 3224-3232."

[0059] Method 2: The method proposed by Xue et al., reference "Xue T, Chen B, Wu J, et al. Video enhancement with task-oriented flow [J]. International Journal of Computer Vision, 2019, 127: 1106-1125."

[0060] Method 3: The method proposed by Wang et al., reference “Wang L, Guo Y, Lin Z, et al. Learning for video super-resolution through HR optical flow estimation [C] / / Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part I 14. Springer International Publishing, 2019: 514-529.”

[0061] Method 4: Tian et al., reference “Tian Y, Zhang Y, Fu Y, et al. Temporally-deformable alignment network for video super-resolution. In 2020 IEEE [C] / / CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 3357-3366.”

[0062] Method 5: The method proposed by Haris et al., reference "Haris M, Shakhnarovich G, Ukita N. Recurrent back-projection network for video super-resolution [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 3897-3906."

[0063] Method 6: The method proposed by Wang et al., reference “Wang X, Chan KCK, Yu K, et al. Edvr: Video restoration with enhanced deformable convolutional networks[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition workshops. 2019:0-0.”

[0064] Method 7: The method proposed by Chan et al., reference “Chan KCK, Wang X, Yu K, et al. Basicvsr: The search for essential components in video super-resolution and beyond[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 4947-4956.”

[0065] Method 8: The method proposed by Chan et al., reference “Chan KCK, Zhou S, Xu X, et al. Basicvsr++: Improving video super-resolution with enhanced propagation and alignment[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 5972-5981.”

[0066] Method 9: The method proposed by Chen et al., reference "Chen P, Yang W, Wang M, et al. Compressed domain deep video super-resolution[J]. IEEE Transactions on Image Processing, 2021, 30: 7156-7169."

[0067] The contents of the comparative experiment are as follows:

[0068] Method 1, method 2, method 3, method 4, method 5, method 6, method 7, method 8, and method 9 are retrained with the same training set, and compressed video super-resolution is performed on the test set, and compared with this method. It is worth noting that methods 1-9 need to train specific models for different encoding configurations and QPs, while this method only needs to train one model. In this experiment, super-resolution is performed on low-resolution compressed videos with QP=22, 27, 32, and 37 in LDB mode. Table 1 shows the comparison of the average PSNR (Peak Signal to Noise Ratio) and SSIM (Structure Similarity Index) of the reconstruction results of each method on the test set. The larger the PSNR (dB) and SSIM values, the better the reconstructed video quality. The best results obtained in all experiments are marked in bold. In addition, "BasketballDrive(LDB,QP=27)", "BQTerrace(RA,QP=32)", "Johnny(RA,QP=27)", "KristenAndSara(LDB,QP=22)", and "PeopleOnStreet(RA,QP=27)" were selected for visual comparison. Figure 5 As shown, Figure 5 (a) represents the original high-resolution video frame, Figure 5 (b) Figure 5 (c) are the image blocks selected for display on the original high-resolution frame and their corresponding low-resolution image blocks after decoding. Figure 5 (d) Figure 5 (e) Figure 5 (f) Figure 5 (g) Figure 5 (h) and Figure 5 (i) Comparison diagram of subjective visual effects after super-resolution of method 1, method 5, method 6, method 7, method 8 and the present invention.

[0069] Table 1

[0070]

[0071] From the experimental results we can see that:

[0072] Methods 1 to 8 are video super-resolution algorithms, which have achieved good results after retraining with compressed videos. Method 9 uses coding priors, which is a good improvement over methods 1 to 7. The present invention makes full use of coding priors, and only needs to deploy one model to achieve superiority over methods 1 to 9 in objective indicators of different QPs. In addition, compared with methods 1, 5, 6, 7, and 8, the reconstruction results of the QP adaptive compressed video super-resolution method based on compression priors proposed in the present invention are visually pleasing and have clearer texture structures.

[0073] In summary, the video reconstructed by the present invention has obvious advantages in subjective visual effects and objective evaluation indicators. Therefore, the present invention is an effective compressed video super-resolution method.

Claims

1. A QP adaptive compression video super-resolution method based on compression prior, characterized by The implementation process includes the following steps: Step 1: Multi-scale feature extraction: Specifically, the feature extraction block composed of multi-scale convolution is used to extract the multi-scale shallow features of the compressed low-resolution video frame and its corresponding compressed prior information (prediction signal, block map, residual); Step 2: Construct a QP modulation module: Specifically, this module takes the frame-level QP as input, generates a modulation map through a 1×1 convolutional layer and a leaky ReLU layer to form a modulation unit, uses the modulation map to recalibrate the input features on the channel, and outputs the modulated features; Step 3: Construct a priori fusion module: Specifically, an adaptive multi-scale prior information fusion module is constructed based on the multi-scale convolution and QP modulation modules. Multiple multi-scale prior information fusion modules are stacked to form a prior information fusion module, which fuses the multi-scale features of the video space and the multi-scale features of the corresponding prior information, and outputs the fused features; Step 4: Bidirectional propagation alignment: Specifically, the existing bidirectional propagation alignment module is used to fuse the fused spatial features in the time domain, and perform feature alignment to output the aligned features; Step 5: Construct a reconstruction module: Specifically, first construct an adaptive enhancement modulation module based on the QP modulation module. The adaptive enhancement modulation modules are stacked to form a reconstruction module. Under the guidance of the frame-level QP, the module adaptively enhances the features. Step 6: Combine the feature extraction block, prior fusion module, bidirectional propagation alignment module and reconstruction module into the final QP adaptive network; Step 7: Use the training data to train the network in step 6; Step 8: During testing, the low-resolution compressed video and its corresponding compression prior (frame-level QP, prediction signal, block map, residual) are used as the input of the network to output the final super-resolution video.

2. The QP adaptive compression video super-resolution method based on compression prior according to claim 1 is characterized in that The QP modulation module in step 2; specifically, given the feature map As input, a CLC structure and a sigmoid layer are used to generate a guidance map G related to the feature content. The CLC structure consists of two 1×1 convolutional layers and one Leakly ReLU layer. The frame-level QP is fed into the modulation unit to generate a 1D modulation map. The modulation process can be summarized as: where F M is the characteristic after modulation by the QP modulation module, f CLC (·) represents the function of CLC structure, Represents element-by-element multiplication, M = f MU (QP),f MU (·) represents the function of the modulation unit, G = σ(f CLC (F)),σ represents the sigmoid function.

3. The QP adaptive compression video super-resolution method based on compression prior according to claim 1 is characterized in that The prior fusion module in step 3; specifically, the prior fusion module consists of m adaptive multi-scale prior information fusion modules and feature fusion blocks. For the i-th adaptive multi-scale prior information fusion module, the input is the high-scale feature generated by the fusion of the modules in the previous stage. and low-scale features and high-scale features extracted on the compressed prior and low-scale features As well as the frame-level QP, the features of different scales are first connected in the channel dimension to achieve preliminary fusion, and we get Then, multi-scale convolution is used to realize the communication and aggregation between multi-scale features. The process can be expressed as: here are respectively the high-scale features and low-scale features after aggregation, f(·,W) represents the convolution with parameter W, δ refers to the Leakly ReLU function, upsample(·,s) represents the nearest neighbor interpolation operation with upsampling parameter s, and pool(·,k) represents the average pooling operation with k×k and step size k. Then, the three-dimensional attention structure is used to further enhance the representation ability of multi-scale features, and the energy function E is used to guide the generation of the three-dimensional attention weights. The features are re-adaptively calibrated according to the importance of each position in the feature map, and then adaptively fused with the high and low features of the input through skip connections. This process can be summarized as follows: Here Respectively represent the high- and low-scale features after adaptive fusion, are 4 additional learnable parameters initialized to 1, It is then fed into the multi-scale QP modulation module as shown in Figure 1(c) together with the frame-level QP; the output of the final adaptive multi-scale prior information fusion module is: Here f MSQPMM (·) represents the function of the multi-scale QP modulation module.

4. The video decompression method based on multi-scale spatiotemporal network according to claim 1 is characterized in that The reconstruction module in step 5 enhances the deep features. Specifically, the reconstruction module consists of n adaptive enhancement modulation modules as shown in Figure 4. The adaptive enhancement modulation module first uses a wide activation structure to enhance the feature flow, that is, the input feature is transformed into the deep feature by a 3×3 convolution layer before the activation layer. The channels are expanded from C to s×C, and then a 3×3 convolution layer is used to adjust the number of feature channels after activation to C, and we get Then, global average pooling, Leaky ReLU layer and sigmoid layer are used to generate attention weights of different channels to adaptively recalibrate the feature F. WA ; The adaptive residual combines the initial features and the recalibrated features through learnable parameters χ, μ to obtain adaptively enhanced features. Finally, the QPMM module is embedded at the tail, and the modulated features and initial features are adaptively fused through learnable parameters γ and η to obtain the adaptive enhanced modulation feature F EMM : F EMM =γ×(f QPMM (C 1×1 (F E )))+η×F in Among them, f QPMM (·) represents the function of QPMM, C 1×1 Represents a 1×1 convolution operation.

Images