Video decompression method based on multi-scale interactive communication space-time network

Abstract

The application discloses a video decompression method based on a multi-scale interactive communication space-time network, mainly comprising the following steps: constructing a multi-scale space-time feature alignment network module, performing implicit alignment on input reference frame sequences and target frames in a multi-scale feature space to obtain aligned multi-scale features; constructing a multi-scale feature enhancement network module, enhancing the multi-scale features obtained in the previous step to obtain enhanced features; constructing a source feature selection enhancement module with the target frame and the enhanced features as inputs; the multi-scale space-time feature alignment network module, the multi-scale feature enhancement network module and the source feature selection enhancement module jointly constitute a multi-scale space-time network, the multi-scale space-time network is uniformly trained by using a data set, and finally a video with compression effects removed is output. The video decompression method based on the multi-scale space-time network can obtain good subjective and objective effects. Therefore, the application is an effective video decompression method.

Description

Technical Field

[0001] This invention relates to compressed video post-processing technology, specifically to a video decompression method based on a multi-scale interactive communication spatiotemporal network, belonging to the field of image / video processing. Background Technology

[0002] Video compression techniques are commonly used to reduce temporal and spatial redundancy in videos for better storage and transmission. However, currently used video compression algorithms (such as H.264 and H.265) inevitably introduce various compression artifacts (such as blocking and ringing) into compressed videos. Especially at low bitrates, compression artifacts significantly reduce video quality, severely impacting user experience and affecting the accuracy of subsequent advanced visual tasks. Video decompression aims to reconstruct high-quality videos from low-quality compressed videos and is a major research hotspot in the field of image / video processing. Video decompression not only enhances the visual quality of compressed videos but also improves the performance of subsequent visual tasks (such as recognition, detection, and tracking).

[0003] Video decompression methods mainly fall into three categories: single-frame decompression methods, single-frame video decompression methods, and multi-frame video decompression methods. Single-frame decompression methods are primarily designed to enhance JPEG images; these methods adapt to the video by reconstructing each individual frame. Single-frame video decompression methods utilize only the high-quality frames in between the current video frame; these methods do not consider the temporal correlation between video frames and are suitable for intra-frame coding modes. Multi-frame video decompression methods use multiple adjacent frames of the target frame as reference frames, reconstructing a high-quality video frame using both the target frame and reference frames. These methods utilize the temporal information of the video and achieve better reconstruction results for compressed videos with inter-frame coding modes. Current video decompression methods do not fully exploit the rich multi-scale information within video frames, thus limiting their performance. Summary of the Invention

[0004] The purpose of this invention is to propose an effective method for decompressing compressed videos.

[0005] The proposed video decompression method based on multi-scale interactive communication spatiotemporal network in this invention includes the following steps:

[0006] (1) Construct a multi-scale spatiotemporal feature alignment module: Specifically, a basic multi-scale residual block is formed by multi-scale convolution and multi-scale feature interaction communication module. The multi-scale residual block and deformable convolution are used to construct a multi-scale spatiotemporal feature alignment module to align the input reference frame and target frame in the feature space.

[0007] (2) Construct a multi-scale feature enhancement module: Specifically, a multi-scale feature enhancement module is constructed using multi-scale residual blocks. The output of the multi-scale spatiotemporal feature alignment module is used as the input of this module to enhance and initially fuse multi-scale features, and output the enhanced depth features.

[0008] (3) Construct source feature selection enhancement module: Specifically, the feature selection branch uses the statistical information of the shallow features of the input target frame to generate an importance map to rescale the enhanced deep features, while the feature retention branch adaptively retains the important information of the deep features;

[0009] (4) Combine the multi-scale spatiotemporal feature alignment module, the multi-scale feature enhancement module and the source feature selection module into the final multi-scale spatiotemporal network;

[0010] (5) Train the network in step (4) using the training data;

[0011] (6) During testing, compressed video is used as the input to the network, and the output is a high-quality video with the compression effect removed. Attached Figure Description

[0012] Figure 1 This is an overview of the decompression method based on multi-scale interactive communication spatiotemporal network of the present invention: where (a) is a block diagram of the multi-scale interactive communication spatiotemporal network structure, (b), (c), and (d) are multi-scale convolutions, and (e) is a structural diagram of the source feature selection enhancement module.

[0013] Figure 2 This is a structural diagram of the multi-scale feature interaction communication module of the present invention.

[0014] Figure 3 These are subjective visual effect comparison images of the test video "FourPeople" compressed to the HEVC standard of QP=37 and then decompressed using the present invention and four other methods: (a) is the test frame, (b) and (e) are the visual effect images after compression and without compression, respectively, and (c), (d), (f) and (g) are subjective visual effect comparison images of the decompression effect of method four, method five, method six and the present invention, respectively. Detailed Implementation

[0015] The present invention will be further described below with reference to the accompanying drawings:

[0016] Figure 1 The video decompression method based on multi-scale spatiotemporal networks includes the following steps:

[0017] (1) Construct a multi-scale spatiotemporal feature alignment module: Specifically, a basic multi-scale residual block is formed by multi-scale convolution and multi-scale feature interaction propagation module. The multi-scale residual block and deformable convolution are used to construct a multi-scale spatiotemporal feature alignment module to align the input reference frame and target frame in the feature space.

[0018] (2) Construct a multi-scale feature enhancement module: Specifically, a multi-scale feature enhancement module is constructed using multi-scale residual blocks. The output of the multi-scale spatiotemporal feature alignment module is used as the input of this module to enhance and initially fuse multi-scale features, and output the enhanced depth features.

[0019] (3) Construct source feature selection enhancement module: Specifically, the feature selection branch uses the statistical information of the shallow features of the input target frame to generate an importance map to rescale the enhanced deep features, while the feature retention branch adaptively retains the important information of the deep features;

[0020] (4) Combine the multi-scale spatiotemporal feature alignment module, the multi-scale feature enhancement module and the source feature selection module into the final multi-scale spatiotemporal network;

[0021] (5) Train the network in step (4) using the training data;

[0022] (6) During testing, compressed video is used as the input to the network, and the output is a high-quality video with the compression effect removed.

[0023] Specifically, in step (1), such as Figure 1 As shown in (a), our constructed multi-scale spatiotemporal feature alignment module includes m multi-scale residual blocks and deformable convolutions. The input compressed video frame sequence is defined as χ... t ={X t-r ,…,X t ,…,X t+r}, where r represents the radius of the time neighborhood, and χ t By inputting the multi-scale spatiotemporal feature alignment module, the aligned multi-scale features can be obtained. The entire process can be represented as:

[0024]

[0025]

[0026] in, and p represents the high-frequency and low-frequency features after spatiotemporal feature alignment, respectively. h and p l They represent X respectively i and Any spatial location in It is Xi The video frame after double cubic downsampling by a factor of 2, where K represents the convolution kernel size, and here K = 3. Let p represent the convolution kernels of the i-th channel in the high-frequency component and the low-frequency component, respectively. k This represents the standard sampling offset, where p k ∈{(-1,-1),(-1,0),…(1,1)},Δ=(Δ h ,Δ l ) and Δm=(Δm h ,Δm l ) represent deformable offsets and modulation scalars at different learnable scales, obtained through network prediction.

[0027] The multi-scale residual block mentioned in step (1) is the basic module constituting the entire network. Each multi-scale residual block contains a multi-scale convolution with C channels, such as... Figure 1 As shown in (c), the system consists of a Rectified Linear Unit (ReLU), two 3×3 ordinary convolutions with 0.5C channels each, and a multi-scale feature interaction communication module. The multi-scale residual block in the multi-scale spatiotemporal feature alignment module has C1 channels.

[0028] Figure 2 The diagram illustrates the structure of the multi-scale feature interaction communication module. Firstly, low-scale features are... Upsampling by 2 times and high-scale features Summing the elements yields Next, using the global information from the Global Average Pooling (GAP) encoding features, statistical information between channels is generated. Subsequently, g is subjected to a 1×1 convolution operation to obtain a compact feature representation. d = max(0.5C / 8, 4); Then, the feature representation z flows through two parallel convolutional layers, each with a kernel size of 1×1, restoring the channel dimension to 0.5C, resulting in the feature representations of the two branches. The attention weights are generated by normalizing the data using the Softmax function.

[0029]

[0030] Finally, attention weights are used to rescale the input multi-scale features to obtain the multi-scale features enhanced by interactive communication.

[0031]

[0032] The multi-scale feature enhancement module built in step (2) is mainly composed of n multi-scale residual blocks, where the number of channels in the multi-scale residual block is C2.

[0033] The source feature selection enhancement module built in step (3) is as follows: Figure 1 As shown in (e), the source feature selection enhancement module consists of a feature selection branch and a feature preservation branch. First, the feature selection branch extracts shallow features from the input target frame using a CRC network module, where the CRC consists of two convolutional layers and one ReLU. Then, it models the attention weights for each neuron in the shallow feature map using the energy function E and the sigmoid activation function, subsequently processing the input features F... EF Scaling is performed according to attention weights, and then a CRC module is used to enhance the scaled features. This process can be expressed as:

[0034]

[0035] Where F FSB f represents the output feature of the feature selection branch. CRC This represents the function of the CRC module. The energy function E is calculated using Yang's method, as referenced in "Yang L, Zhang RY, Li L, et al. Simam: A simple, parameter-free attention module for convolutional neural networks[C] / / International Conference on Machine Learning.PMLR,2021:11863-11874.".

[0036] Simultaneously, the feature-preserving branch adaptively preserves the input features through a CRC module to obtain F. FRB Finally, the output features of the feature retention branch and the output features of the feature selection branch are added together and passed through a CRC module to obtain the final enhanced feature F. SFSE The entire process can be represented as:

[0037] F SFSE =f CRC (f FSB (F EF ,X t )+f CRC (F EF )),

[0038] Where f FSBThis represents the function for feature selection branches. Except for the last CRC module, whose output channel is 1, the input and output channels of the remaining CRC modules are both 0.5C2.

[0039] In step (4), the multi-scale spatiotemporal network is composed of the multi-scale spatiotemporal feature alignment module in step (1), the multi-scale enhancement module in step (2), and the source feature selection enhancement module in step (3). They are trained uniformly, and the loss function of the multi-scale spatiotemporal network is expressed as:

[0040]

[0041] in Y represents the high-quality frame reconstructed by the network, and Y represents the original video frame that has not been compressed.

[0042] During training in step (5), the hyperparameters r, C1, m, C2, and n were set to 3, 48, 5, 64, and 10, respectively. All videos from the training, test, and validation sets were compressed using HEVC's HM16.5 software in LDP mode at four different QP values: 27, 32, 37, and 42. During training, 128×128 blocks were randomly cropped from the original videos and their corresponding compressed videos, and data augmentation was performed using flipping and rotation, with a batch size of 32. The default ADMM was used as the optimizer, with a learning rate of 0.0001, which remained constant throughout.

[0043] To verify the effectiveness of this invention, comparative experiments were conducted on 18 standard test sequences from the Joint Collaborative Video Coding Cooperation (JCT-VC). In these experiments, the method of this invention was compared with two typical image decompression algorithms and four video decompression algorithms. These six algorithms are:

[0044] Method 1: The method proposed by Dong et al., see reference "Dong C, Deng Y, Loy CC, et al. Compression artifacts reduction by a deep convolutional network[C]. IEEE International Conference on Computer Vision. 2015:576-584."

[0045] Method 2: The method proposed by Zhang et al., see reference "Zhang K, Zuo W, Chen Y, et al. Beyonda gaussian denoiser: Residual learning of deep CNN for image denoising[J]. IEEE Transactions on Image Processing, 2017, 26(7):3142-3155."

[0046] Method 3: The method proposed by Yang et al., see reference "Yang R, Xu M, Wang Z, et al. Multi-frame quality enhancement for compressed video[C]. IEEE Conference on ComputerVision and Pattern Recognition. 2018:6664-6673."

[0047] Method 4: The method proposed by Guan et al., see reference "Guan Z, Xing Q, Xu M, et al. MFQE 2.0: A New Approach for Multi-Frame Quality Enhancement on Compressed Video[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(3): 949-963."

[0048] Methods 5 and 6: The methods proposed by Deng et al., referenced in “Deng J, Wang L, Pu S, et al. Spatio-temporal deformable convolution for compressed video quality enhancement[C]. AAAI Conference on Artificial Intelligence. 2020, 34(07): 10696-10703.” Methods 5 and 6 represent the simple and complex models in this paper, respectively.

[0049] The comparative experiment is as follows:

[0050] Decompression was performed on the test set using methods 1, 2, 3, 4, 5, 6, and the method of this invention, respectively. In this experiment, compressed videos with QP = 27, 32, 37, and 42 were decompressed. Table 1 shows a comparison of ΔPSNR (Peak Signal to Noise Ratio) and ΔSSIM (Structure Similarity Index) of the reconstruction results of each method, ΔPSNR (dB) and ΔSSIM (×10) -2 A higher value indicates better reconstructed video quality; the best results obtained from all experiments are marked in bold. Additionally, for visual comparison, a single frame from the "FourPeople" sequence is provided. The compressed video frames of "FourPeople" are shown below. Figure 3 As shown in (a). Figure 3 (b) Figure 3 (e) Visual representations of the compressed and uncompressed versions, respectively. Figure 3 (c) Figure 3 (d) Figure 3 (f) and Figure 3 (g) are subjective visual effect comparison diagrams of method 4, method 5, method 6 and the present invention for decompression effect.

[0051] Table 1

[0052]

[0053]

[0054] The experimental results show that:

[0055] Methods 1 and 2 are image decompression methods. When used for video decompression, they do not consider the temporal correlation between adjacent frames, resulting in poor performance. Methods 3, 4, 5, 6, and this invention are video decompression methods that reconstruct a high-quality target frame using the target frame and its surrounding adjacent frames, achieving better results than methods 1 and 2. Compared to methods 3, 4, 5, and 6, the video decompression method based on a multi-scale spatiotemporal network proposed in this invention achieves the best objective metrics and better visual effects on the test set.

[0056] In summary, the video reconstructed by this invention has significant advantages in both subjective visual effects and objective evaluation metrics. Therefore, this invention is an effective video decompression method.

Claims

1. A video decompression method based on a multi-scale interactive communication spatiotemporal network, characterized in that: The video decompression refers to the process of repairing and reconstructing artifacts introduced during quantization, prediction, and encoding of compressed video using a deep neural network model after compression. The implementation of this method includes the following steps: Step 1: Construct a multi-scale spatiotemporal feature alignment module: Specifically, a basic multi-scale residual block is formed by multi-scale convolution and multi-scale feature interaction communication module. The multi-scale residual block and deformable convolution are used to construct a multi-scale spatiotemporal feature alignment module to align the input reference frame and target frame in the feature space. Step 2: Construct a multi-scale feature enhancement module: Specifically, a multi-scale feature enhancement module is constructed using multi-scale residual blocks. The output of the multi-scale spatiotemporal feature alignment module is used as the input of this module. The aligned multi-scale features are enhanced and initially fused, and the enhanced depth features are output. The multi-scale residual block consists of a single channel with a certain number of channels. C The multi-scale convolution uses one rectified linear unit (ReLU) and two channels with a total of 0.5 channels. C of It consists of a regular convolution and a multi-scale feature interaction communication module; The multi-scale feature interaction communication module first transmits low-scale features... Upsampling by 2 times and high-scale features Summing the elements yields Next, using the global information from the Global Average Pooling (GAP) encoding features, statistical information between channels is generated. ; then, pass The convolution operation yields a compact feature representation. , Then, feature representation z Flowing through two parallel convolutional layers, each with a 1×1 kernel size, restores the channel dimension to 0.

5. C The characteristic representations of the two branches are obtained as follows: , The Softmax function is used for normalization to generate the inter-channel relative attention weight matrix: ， Finally, attention weights are used to rescale the input multi-scale features to obtain the multi-scale features enhanced by interactive communication. The data is then input into subsequent modules for further feature fusion or enhancement processing. ； Step 3: Construct the source feature selection and enhancement module: The source feature selection and enhancement module includes a feature selection branch and a feature retention branch. The feature selection branch generates a channel weighting coefficient matrix based on the statistical information of the shallow features of the input target frame, and performs channel weighting and scaling on the enhanced deep features; the feature retention branch retains the key response information of the deep features; the outputs of the two branches are fused in the channel dimension to form the final enhanced features; Step 4: Connect the multi-scale spatiotemporal feature alignment module, the multi-scale feature enhancement module, and the source feature selection enhancement module in sequence to form the final multi-scale interactive communication spatiotemporal network, wherein the output of the multi-scale feature enhancement module is used as the input of the source feature selection enhancement module; Step 5: Use the training dataset to train the network from Step 4; Step 6: During testing, the compressed video is used as the network input, and the final output is a high-quality video with the compression effect removed.

2. The video decompression method based on a multi-scale interactive communication spatiotemporal network according to claim 1, characterized in that... The multi-scale spatiotemporal feature alignment module in step one uses deformable convolution to align the reference frame and the target frame in the feature space at different scales.

3. The video decompression method based on a multi-scale interactive communication spatiotemporal network according to claim 1, characterized in that... The source feature selection and enhancement module in step three enhances the deep features; The source feature selection enhancement module consists of a feature selection branch and a feature preservation branch. The feature selection branch uses a CRC network module to extract shallow features from the input target frame, where the CRC consists of two convolutional layers and one ReLU layer; then, it uses an energy function... E The sigmoid activation function is used to generate a channel-weighted coefficient matrix with values ​​ranging from (0,1). This matrix is ​​used to independently model each channel, enabling adaptive channel enhancement and reflecting the absolute importance of channel features. This matrix serves as the channel-independent attention weight matrix. Subsequently, the input deep features are... The scaling is performed channel-by-channel according to attention weights, and then a CRC module is used to enhance the scaled features. This process can be expressed as: ， in This represents the output feature of the feature selection branch. This represents a function of the CRC module; simultaneously, the feature-preserving branch adaptively preserves the input features through a CRC module, resulting in... Finally, the output features of the feature retention branch and the output features of the feature selection branch are added together and passed through a CRC module to obtain the final enhanced features. .

Images