Compressed video quality improvement method based on CNN-Transform hybrid model
A CNN-Transformer hybrid model addresses the limitations of CNN-based methods by integrating and enhancing spatial and temporal information, resulting in improved compressed video quality through a U-net network and deformable convolution layers.
Patent Information
- Application Number
- CN202310697897.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-13
- Publication Date
- 2024-12-13
AI Technical Summary
Existing video compression algorithms, particularly CNN-based methods, struggle with limited receptive fields, making it difficult to model long-range dependencies and effectively improve compressed video quality due to the inherent limitations of convolution processes.
A hybrid model combining CNN and Transformer architectures is employed, utilizing a U-net network and deformable convolution layers to integrate spatial and temporal information, followed by Transformer blocks to enhance feature extraction, and a stack of convolution layers to refine and reconstruct the final enhanced frame.
The hybrid model significantly improves video quality by enhancing spatial and temporal features, resulting in higher PSNR values compared to traditional methods, demonstrating superior performance in reconstructing compressed videos.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] The present invention relates to a compressed video quality improvement technology, and in particular to a compressed video quality improvement method based on a CNN-Transformer hybrid model, and belongs to the field of image communication. Background Art
[0002] With the rapid development of the multimedia industry, consumers have higher and higher requirements for video quality. Due to the demand for long videos, short videos, live broadcasts and other video streams, a large amount of video data needs to be transmitted through the Internet with limited bandwidth every day. In this regard, many video compression algorithms have been proposed, such as HEVC. HEVC compresses and encodes videos based on blocks, and various compression artifacts are introduced during compression, which significantly reduces the video quality and affects the user's viewing experience. At present, the mainstream compressed video quality improvement methods are mainly divided into loop filtering methods and post-processing methods. Among them, the loop filtering method acts on the inside of the video codec. Limited by the overall complexity requirements of the codec, the loop filtering method cannot be designed to be too complicated, and the performance improvement is limited. The post-processing method is to improve the quality of the compressed video after the video codec standard reconstructs it. Therefore, it does not change the video codec structure, has fewer restrictions, and has relatively better performance.
[0003] With the development of deep learning, CNN-based methods have been widely used to enhance the quality of compressed videos and have achieved remarkable results. These CNN algorithms can be divided into single-frame methods and multi-frame methods according to the number of frames used. Single-frame algorithms process each video frame individually without utilizing the temporal information of adjacent video frames. Multi-frame algorithms improve this by utilizing both spatial and temporal information of the video. However, due to the inherent limitations of CNN, the receptive field is limited during the convolution process, making it difficult to model long-range dependencies on relevant information. The Transformer model, which has emerged in recent years, has a unique self-attention mechanism that can effectively obtain global information.
[0004] Therefore, the present invention intends to study a new compressed video quality improvement algorithm that combines the respective characteristics of CNN and Transformer. Summary of the invention
[0005] In view of the shortcomings of CNN in improving the quality of compressed videos, the present invention aims to propose a method for improving the quality of compressed videos based on a CNN-Transformer hybrid model.
[0006] The present invention proposes a method for improving the quality of compressed videos based on a CNN-Transformer hybrid model, which mainly includes the following steps: (1) The compressed video sequence is input into the CNN-based feature initial processing module to obtain feature frames that integrate spatial and temporal information.
[0007] (2) The feature frame obtained in (1) is input into the Transformer-based feature depth processing module to obtain a further aligned and fused feature frame.
[0008] (3) The output of (2) is input into the reconstruction module, and finally the compressed video sequence with improved quality is obtained.
[0009] Specifically, in step (1), it is assumed that a compressed video is ,Depend on n Frame composition, that is When the promotion t ( t ∈[1,n]) target frame at time When the quality is , select the first two adjacent frames , With the next two frames , As a reference frame, to obtain time domain reference information. Perform the "Contcatenate" operation on the channel dimension. This process is represented by formula (1-1).
[0010]
[0011] Then will Input Figure 1 The feature preliminary processing module shown in FIG. 1 includes a U-net network, a deformable convolution layer, and a conventional convolution layer. The structure of the U-net network is as follows: Figure 2 As shown, its role is to predict the motion offset between adjacent frames At the beginning of the U-net network, we use a regular convolutional layer to In addition, the network mainly consists of three consecutive downsampling convolutional layers and three consecutive upsampling deconvolutional layers. The purpose of downsampling and upsampling is to expand the receptive field of the convolutional layer through multi-scale so as to obtain features faster. Finally, a regular convolutional layer is used to generate the predicted motion offset. In the U-net network, the step size of the regular convolution layer is set to 1 and the convolution size is , and the linear rectifier unit (RELU) is used as the activation function. The convolution step size of the downsampling layer is 2, and the convolution size is The convolution step size of the upsampling layer is 2, and the convolution size is In addition, the output features of the same size in the downsampling layer and the upsampling layer are spliced through jump connections to reuse features and speed up the convergence of the network. The working process of U-net is shown in formula (1-2).
[0012]
[0013] in, Represents a U-net network.
[0014] As shown in Figure 1, after the U-net network, the convolution kernel size is , a deformable convolutional layer with a stride of 1 is used for further input processing.
[0015]
[0016]
[0017] in, It represents the fused feature map output by the feature preliminary processing module. Represents the deformable convolution process.
[0018] Subsequently, a regular convolution layer with a convolution kernel size of 3x3 is used for further processing. The above process can be expressed by formula (1-4).
[0019]
[0020] in Represents the enhanced features obtained after processing through the regular convolutional layer. Represents the working process of a conventional convolutional layer.
[0021] Then, as shown in step (2), Input into the feature depth processing module. The function of this module is described as follows.
[0022] The Transformer block structure in the feature depth processing module is as follows Figure 3 shown.
[0023] Combination Figure 1 and Figure 3 , is the output of the nth SwinV2-Transformer of the mth Transformer block.
[0024]
[0025] in, Represents the action process of the nth SwinV2-Transformer in the mth Transformer block.
[0026] The output of the Transformer block is shown in formula (1-6).
[0027]
[0028] in, represents the input of the mth Transformer block, Represents the working process of the convolutional layer in the mth Transformer block.
[0029] For the Transformer block, the input and output of each layer of transformer can be expressed as follows:
[0030] in Represents the action process of the mth Transformer block.
[0031] The output of the feature depth processing module can be expressed as follows:
[0032] in Represents the working process of the convolutional layer after the stacked Transformer blocks.
[0033] Finally, as described in step (3), we will and Add As the input of the reconstruction module. Our reconstruction module consists of a stack of convolutional layers with a kernel size of 3x3 and a stride of 1. Except for the last layer, each convolutional layer is followed by a linear rectifier unit as an activation function. The working process of the reconstruction module is as follows:
[0034]
[0035] in Represents the process of the reconstruction module. represents the residual of the generated enhanced frame, Represents the final enhanced frame. Aiming at the shortcomings of CNN in improving the quality of compressed videos, the present invention aims to propose a method for improving the quality of compressed videos based on a CNN-Transformer hybrid model.
[0036] The present invention proposes a method for improving the quality of compressed videos based on a CNN-Transformer hybrid model, which mainly includes the following steps: (1) The compressed video sequence is input into the CNN-based feature initial processing module to obtain feature frames that integrate spatial and temporal information.
[0037] (2) The feature frame obtained in (1) is input into the Transformer-based feature depth processing module to obtain a further aligned and fused feature frame.
[0038] (3) The output of (2) is input into the reconstruction module, and finally the compressed video sequence with improved quality is obtained.
[0039] Specifically, in step (1), it is assumed that a compressed video is ,Depend on n Frame composition, that is When the promotion t ( t ∈[1,n]) target frame at time When the quality is , select the first two adjacent frames , With the next two frames , As a reference frame, to obtain time domain reference information. Perform the "Contcatenate" operation on the channel dimension. This process is represented by formula (1-1).
[0040]
[0041] Then will Input Figure 1 The feature preliminary processing module shown in FIG. 1 includes a U-net network, a deformable convolution layer, and a conventional convolution layer. The structure of the U-net network is as follows: Figure 2 As shown, its role is to predict the motion offset between adjacent frames At the beginning of the U-net network, we use a regular convolutional layer to In addition, the network mainly consists of three consecutive downsampling convolutional layers and three consecutive upsampling deconvolutional layers. The purpose of downsampling and upsampling is to expand the receptive field of the convolutional layer through multi-scale so as to obtain features faster. Finally, a regular convolutional layer is used to generate the predicted motion offset. In the U-net network, the step size of the regular convolution layer is set to 1 and the convolution size is , and the linear rectifier unit (RELU) is used as the activation function. The convolution step size of the downsampling layer is 2, and the convolution size is The convolution step size of the upsampling layer is 2, and the convolution size is In addition, the output features of the same size in the downsampling layer and the upsampling layer are spliced through jump connections to reuse features and speed up the convergence of the network. The working process of U-net is shown in formula (1-2).
[0042]
[0043] in, Represents a U-net network.
[0044] As shown in Figure 1, after the U-net network, the convolution kernel size is , a deformable convolutional layer with a stride of 1 is used for further input processing.
[0045]
[0046]
[0047] in, It represents the fused feature map output by the feature preliminary processing module. Represents the deformable convolution process.
[0048] Subsequently, a regular convolution layer with a convolution kernel size of 3x3 is used for further processing. The above process can be expressed by formula (1-4).
[0049]
[0050] in Represents the enhanced features obtained after processing through the regular convolutional layer. Represents the working process of a conventional convolutional layer.
[0051] Then, as shown in step (2), Input into the feature depth processing module. The function of this module is described as follows.
[0052] The Transformer block structure in the feature depth processing module is as follows Figure 3 shown.
[0053] Combination Figure 1 and Figure 3 , is the output of the nth SwinV2-Transformer of the mth Transformer block.
[0054]
[0055] in, Represents the action process of the nth SwinV2-Transformer in the mth Transformer block.
[0056] The output of the Transformer block is shown in formula (1-6).
[0057]
[0058] in, represents the input of the mth Transformer block, Represents the working process of the convolutional layer in the mth Transformer block.
[0059] For the Transformer block, the input and output of each layer of transformer can be expressed as follows:
[0060] in Represents the action process of the mth Transformer block.
[0061] The output of the feature depth processing module can be expressed as follows:
[0062] in Represents the working process of the convolutional layer after the stacked Transformer blocks.
[0063] Finally, as described in step (3), we will and Add As the input of the reconstruction module. Our reconstruction module consists of a stack of convolutional layers with a kernel size of 3x3 and a stride of 1. Except for the last layer, each convolutional layer is followed by a linear rectifier unit as an activation function. The working process of the reconstruction module is as follows:
[0064]
[0065] in Represents the process of the reconstruction module. represents the residual of the generated enhanced frame, Represents the final enhanced frame. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] Figure 1 It is a block diagram of a compressed video quality improvement method based on a CNN-Transformer hybrid model of the present invention.
[0067] Figure 2 It is a structural diagram of the U-net network in the feature initial processing module of the present invention.
[0068] Figure 3 It is a structural diagram of the Transformer block of the feature depth processing module in the present invention.
[0069] Figure 4The following is a comparison of the visual quality of the subjective effects of the HEVC standard, the method of the present invention, and three comparison methods when the PartyScene sequence is at QP=37. Figure (a) is the 6th frame of the original PartyScene sequence, Figure (b) is a magnified local detail of the 6th frame of the sequence after being compressed by the HEVC standard, Figure (c) is a magnified local detail of the same frame after the sequence is compressed by the HEVC standard and then processed by the comparison method [1]. The PSNR of the whole frame is increased by 0.278db compared to the whole frame processed in Figure (b), and Figure (d) is a magnified local detail of the same frame after the sequence is compressed by the HEVC standard and then processed by the comparison method [2]. Figure (e) shows an enlarged view of the local details of the same frame after the sequence is compressed by the HEVC standard and then processed by the comparison method [3]. The PSNR of the whole frame is increased by 0.832db compared with the whole frame processed in Figure (b). Figure (f) shows an enlarged view of the local details of the same frame after the sequence is compressed by the HEVC standard and then processed by the present invention. The PSNR of the whole frame is increased by 1.311db compared with the whole frame processed in Figure (b). Figure (g) shows an enlarged view of the local details of the original frame.
[0070] Figure 5 The following is a comparison of the subjective visual quality of the HEVC standard, the method of the present invention, and three comparison methods when the BasketballDrill sequence is at QP=37. Figure (a) is the 20th frame of the original BasketballDrill sequence, Figure (b) is an enlarged view of the local details of the 20th frame of the sequence after being compressed by the HEVC standard, and Figure (c) is an enlarged view of the local details of the same frame after the sequence is compressed by the HEVC standard and then processed by the comparison method [1]. The PSNR of the whole frame increases by 0.331db compared with the whole frame processed in Figure (b). Figure (d) is an enlarged view of the local details of the same frame after the sequence is compressed by the HEVC standard and then processed by the comparison method [2]. Figure (e) is a magnified image of the local details of the same frame after the sequence is compressed by the HEVC standard and then processed by the comparison method [3]. The PSNR of the whole frame is increased by 0.995db compared with the whole frame processed in Figure (b). Figure (f) is a magnified image of the local details of the same frame after the sequence is compressed by the HEVC standard and then processed by the present invention. The PSNR of the whole frame is increased by 1.204db compared with the whole frame processed in Figure (b). Figure (g) is a magnified image of the local details of the original frame. DETAILED DESCRIPTION
[0071] The present invention is further described in detail below in conjunction with the embodiments. It is necessary to point out that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Technical personnel familiar with the relevant field may make some non-essential improvements and adjustments to the present invention according to the above invention content for specific implementation, which should still fall within the scope of protection of the present invention.
[0072] Figure 1 Specifically, a method for improving the quality of compressed videos based on a CNN-Transformer hybrid model is involved, which can be divided into the following steps: (1) The compressed video sequence is input into the CNN-based feature initial processing module to obtain feature frames that integrate spatial and temporal information.
[0073] (2) The feature frame obtained in (1) is input into the Transformer-based feature depth processing module to obtain a further aligned and fused feature frame.
[0074] (3) Input the output of (2) into the reconstruction module, and finally obtain the compressed video sequence with improved quality. As described in the above steps (1), (2), and (3), the experiment verifies that this algorithm can achieve a good quality enhancement effect on the compressed video.
[0075] In order to better illustrate the effectiveness of the present invention, two test sequences "PartyScene" and "BasketballDrill" are selected to compare with the compression standard HEVC and other methods. The subjective visual effects are as follows: Figure 4 and Figure 5 Table 1 shows the PSNR improvement of the method of the present invention and the comparative method under four representative QPs (27, 32, 37 and 42) compared with the HEVC standard compressed video, in db. This shows that the present invention is superior to the classic convolutional neural network method.
[0076] Table 1 Improvement effect of different methods when QP is 37
[0077] Table 2 The improvement effect of different methods when QP is 27, 32 and 42
[0078] The comparison method is: Method 1: The method proposed by Dong C, Deng Y et al., reference “Compression artifacts reduction by a deep convolutional network[C] / / Proceedings of the IEEE international conference on computer vision. 2015: 576-584.”
[0079] Method 2: The method proposed by Guan Z, Xing Q, Xu M et al., reference “MFQE 2.0: A new approach for multi-frame quality enhancement on compressed video[J]. IEEE transactions on pattern analysis and machine intelligence, 2019, 43(3): 949-963.”
[0080] Method 3: The method proposed by Zhao M, Xu Y et al., reference “Recursive fusion and deformable spatiotemporal attention for video compression artifact reduction[C] / / Proceedings of the 29th ACM international conference on multimedia.2021: 5646-5654.”.
Claims
1. A method for improving the quality of compressed videos based on a CNN-Transformer hybrid model, characterized in that: According to the advantages and disadvantages of CNN and Transformer, an overall framework of a compressed video quality improvement method combining spatial and temporal information is constructed. The steps are as follows: (1) The compressed video sequence is input into the CNN-based feature initial processing module to obtain feature frames that integrate spatial and temporal information; The specific method is to place five adjacent frames They are connected together through the "Concatenate" operation to combine spatial and temporal correlations, and then predicted by a network based on the U-net structure. and arrive The movement offset The network mainly consists of three consecutive downsampling convolutional layers and three consecutive upsampling deconvolutional layers, and finally a regular convolutional layer is used to generate the predicted motion offset. , in the U-net network, the step size of the conventional convolution layer is set to 1 and the convolution size is , ReLU is used as the activation function, the convolution step size of the downsampling layer is 2, and the convolution size is , the convolution step size of the upsampling layer is 2, and the convolution size is , after the U-net network, a convolution kernel size of , a deformable convolution layer with a stride of 1 is combined with the offset to further process the input, and finally a convolution kernel with a size of The conventional convolutional layer processing obtains the feature frame that integrates the spatial and temporal information; (2) The feature frame obtained in (1) is input into the feature depth processing module based on Transformer to obtain a further aligned and fused feature frame; The specific method is to input the feature frame obtained in (1) into the feature deep processing module, and use the adaptability and excellent global information extraction ability of Transformer to perform deep processing on the feature frame to obtain a further fused feature frame; (3) Input the output of (2) into the reconstruction module, and finally obtain the compressed video sequence with improved quality; The reconstruction module consists of a stack of convolution kernels of size The network consists of convolutional layers with a stride of 1. Except for the last layer, each convolutional layer is followed by a linear rectifier unit as an activation function. The input feature frames are further aligned and fused, and the stacked convolutional layers are used to learn spatiotemporal information from the features and reconstruct the residual of the enhanced frame. Finally, they are added to the frame to be enhanced to output the enhanced compressed video frame.