Full-I-frame video compression method based on channel context

Abstract

The invention provides a channel context-based full-I-frame video compression method, which mainly comprises the following steps of: firstly, converting an input frame # imgabs0 # into a potential feature y through an encoder network, and introducing a hyper-prior model to capture hidden information of potential representation so as to assist in parameter generation of an entropy model; then, the information is converted and expressed as new intermediate information # imgabs1 #. Intermediate information # imgabS2 # is quantized and then enters an arithmetic encoder to obtain a bit stream, a super-prior decoding network reconstructs super-prior information to obtain entropy model estimation parameters, the data passes through a decoding network, the decoding network decodes by using the parameters to obtain reconstructed data, the reconstructed data passes through an image enhancement module, and image data reconstructed under deep learning is output. Experimental results show that the method can effectively improve the compression effect, and original information is reserved as far as possible after the image is compressed.

Description

Technical Field

[0001] The invention relates to a video encoding technology, in particular to an all-Intra video compression method based on a channel context, and belongs to the field of image communication. Background Art

[0002] In recent years, the popularity of high-definition video applications and the emergence of new video formats have led to an unprecedented increase in the amount of video data. In addition, thanks to the rapid development of mobile Internet, users can upload high-definition videos anytime and anywhere, which further poses severe challenges to video transmission and storage systems. As people's demand for visual content continues to grow, research on image and video compression has gradually attracted the attention of researchers. The existing coding standards, JPEG, JPEG2000, BPG for image compression and AVC / H.264, HEVC / H.265, VVC / H.266 for video compression, are all block-related, and each block is also executed sequentially. Traditional image and video compression methods achieve good performance through carefully designed artificial techniques and complex processing, but the use of hybrid frameworks to improve compression efficiency introduces a block effect. Although each module is usually designed and optimized independently, this also limits the compression performance due to local optimization and cannot be jointly optimized in an end-to-end manner. In recent years, with the resurgence of deep learning technology and its widespread success in the field of computer vision, video coding technology has opened up a new research field of end-to-end automatic modeling. Different from traditional video coding compression standards, learning-based video compression methods can more efficiently extract and utilize data features while avoiding block artifacts, thereby achieving better reconstructed frame quality.

[0003] Recently, various learning-based image compression models have been proposed and achieved remarkable performance. Most of these models are based on variational autoencoders and follow the process of transformation, quantization, entropy coding, and inverse transformation. Entropy coding plays an important role in improving model performance. A strong and accurate entropy model usually produces fewer bits. State-of-the-art image compression models usually combine hyper-prior models and context to estimate conditional entropy and use conditional probability for encoding. However, the entropy models in these image compression fail to consider the dependencies between different channels in the latent features, which hinders the accurate estimation of the entropy model. Previous studies have focused on image and video compression, but few studies have analyzed learning-based ALL-Intra video coding. Intra-frame compression not only affects the coding quality of the current frame, but also has a critical impact on inter-frame compression because the key frames using intra-frame prediction will be referenced by inter-frame compression. Therefore, the efficiency and accuracy of intra-frame compression are very important.

[0004] Traditional deep video coding methods mainly establish corresponding models by analyzing the content characteristics of deep videos, thereby improving the efficiency of deep video coding. In recent years, with the continuous development and rapid progress of deep learning related theories represented by convolutional neural networks (CNN), the optimal processing method through machine autonomous learning has become an emerging research direction in the field of video coding. Different from traditional image or video coding methods, the method based on convolutional neural networks specifically designs the deep neural network structure according to the specific problems involved, and uses appropriate training strategies to enable the network to complete a specific goal in the coding system. Related studies have shown that methods based on convolutional neural networks can handle more complex tasks and obtain better performance. Inspired by this, studying how to integrate convolutional neural networks into deep video coding systems and improve coding efficiency through their excellent learning characteristics has important research significance and broad application prospects. Summary of the invention

[0005] The main purpose of intra-frame compression is to compress and encode intra-frame coded frames through intra-frame compression algorithms, and generate higher quality decoded frames at a certain bit rate at the decoding end. The present invention first systematically studies and summarizes the key algorithms and improvements of deep learning compression technology, adopts a network structure based on variational autoencoders, and designs a full I-frame video compression framework. In order to solve the problem of image quality loss, a post-processing module is added, and a cascaded residual network is used to reduce the information loss generated by the current frame during the compression process.

[0006] The present invention proposes a full I-frame video compression method based on variational autoencoder and combined with cross-channel context and super prior information. Firstly, by introducing the variational autoencoder structure and adopting the deep residual learning method, different attention mechanisms are used for feature layers at different levels to achieve better feature enhancement effects, and super prior information encoding structure is introduced to achieve sampling that is more in line with the conditional Gaussian distribution of pixels in the image. At the same time, the context network is improved to obtain more accurate probability model parameters of potential information. The overall framework of the present invention is shown in the attached figure. Figure 1 shown.

[0007] The main steps are as follows: (1) Input frame First, it is sent to the main encoder network ( ), x is transformed into latent features y; (2) Combine the latent features y obtained in (1) and quantize and entropy encode them. The quantized latent representation is express; (3) The super prior model is introduced, and y passes through the super prior encoder network ( Processing, the modules in the super prior network obtain implicit information and then transform it into new intermediate information ; (4) Intermediate information Quantified as Then enter the arithmetic encoder to obtain the bit stream, the super prior decoder network ( Reconstruct the hyper-prior information and obtain the estimated parameters of the entropy model; (5) Recover the known probability distribution parameters and then Input to the main decoder network ( ) to generate a reconstructed frame; (6) The reconstructed frame is then passed through the quality enhancement module to output the reconstructed frame under deep learning .

[0008] Specifically, in the transform coding process, the compression process of the input frame can be expressed by formula (1):

[0009] in, and They are the parameters that need to be optimized in the main encoder network and the main decoder network, respectively. Q represents the quantization process, and E represents the quality enhancement module.

[0010] The super prior is introduced to use the side information z to capture the spatial dependency between elements, which is expressed as:

[0011] in and yes and The corresponding optimization parameters are is the model estimate.

[0012] In end-to-end image compression coding based on learning, the context module can be used to optimize the image compression process. However, the existing latent representation entropy model cannot fully utilize its spatial and channel correlation. We consider the correlation between latent feature channels in the entropy model and combine it with the context, and propose a cross-channel context model as shown in the attached figure. Figure 2 As shown in the figure, the performance of the entropy model is improved. First, the latent features Divide into two groups of features by channel and , then the feature Get features through a mask convolution ,feature Generate features by searching the model across channels , the decoded features are combined with the super-prior output, and the masked 2D convolution and cross-channel search modules are used to fuse the super-prior and the autoregressive prior for accurate entropy estimation. The cross-channel search module searches on the latent features of the second group to find the relevant latent features that are most similar to the target latent features in the first group of latent features. We use the inner product to estimate the correlation between the two feature blocks, thereby establishing the correlation between them. The larger the inner product, the greater the correlation between the two feature blocks, and the more feature information can be transferred. Conversely, the smaller the inner product, the greater the correlation between the two feature blocks.

[0013] The encoder network in step (1) consists of two transformation blocks as shown in the following figure. Figure 3 As shown in the figure: residual downsampling layer and residual enhancement layer. The residual downsampling layer includes two layers of convolution. In addition, the front layer uses LeakyReLU as the activation function, and the back layer is GDN. Figure 4 As shown in the figure, a lightweight residual block (LRB) is used to learn the front-layer features and combine them with the attention features. The lightweight residual block has fewer parameters and a comparable receptive field than the usual residual block, which also reduces the difficulty of fitting. Based on the lightweight residual block, a pooling-based attention module is introduced in the middle of the encoding network to enhance the features of the non-local domain. At the end of the encoder, there is a downsampling convolution layer and a lightweight non-local residual attention layer that is different from the pooling-based attention, as shown in the figure. Figure 5 As shown in the figure, the receptive field is increased. The decoding network has a similar structure to the encoding network, which is the inverse process of feature encoding. Figure 3 As shown in (3), the up-sampling module (Up-sample) in the residual up-sampling layer is composed of a convolutional layer and a PixelShuffle up-sampling method in series. This up-sampling method increases the size of the feature map while reducing the number of corresponding channels.

[0014] Inspired by the success of image quality enhancement networks, we add a lightweight Residual Dense Network (RDN) as our post-processing network.

[0015] The rate-distortion function of the training algorithm of the present invention is:

[0016] d(·) represents the reconstructed image and the original image The distortion measurement between . Different distortion functions are used according to the final performance indicators of the model. Peak signal-to-noise ratio (PSNR) is often used as a measurement method for signal reconstruction quality in fields such as image compression. It is often simply defined by mean square error (MSE), and because PSNR and MSE are negatively correlated functionally, when PSNR is used as the image distortion evaluation in rate-distortion evaluation, MSE is used as the distortion term in the loss function. When another image evaluation distortion function, multi-scale structural similarity (MS-SSIM), is used, the higher the MS-SSIM, the better the quality of the two images, so 1-MS-SSIM is used as the distortion term of the rate-distortion loss function. , Respectively represent the estimated bit rate of the potential features and the super prior information after quantization. By adjusting the Lagrangian constant λ, the compressed image under different compression conditions can be obtained. In the present invention, when the distortion term is calculated using MSE, , use MS-SSIM to calculate the distortion term .

[0017] The advantages and beneficial technical effects of the present invention compared with the prior art are as follows: (1) The present invention proposes a channel context-based full I-frame video compression method. The method draws inspiration from the learning-based image compression method and studies the learning-based full I-frame video compression method, fully considering the correlation between different channels of potential features and combining super-prior information for entropy estimation.

[0018] (2) The present invention utilizes two-dimensional mask convolution and the dependency between channels to perform more accurate entropy estimation. In this method, the full I-frame compression and quality enhancement modules are jointly trained and optimized to minimize the loss function.

[0019] (3) Experimental results show that this method can achieve better compression effects both in terms of objective indicators and subjective effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a block diagram of a full I-frame video compression method based on channel context of the present invention.

[0021] Figure 2 It is a network structure diagram based on the cross-channel search model in the present invention.

[0022] Figure 3 2 is a network structure diagram of two transformation blocks in the encoder of the present invention.

[0023] Figure 4 This is a network structure diagram of the pooled attention mechanism in the present invention.

[0024] Figure 5 This is the network structure diagram of the lightweight non-local residual attention mechanism in the present invention.

[0025] Figure 6 The rate-distortion curves of different methods on the Kodak dataset.

[0026] Figure 7 Rate-distortion curves of different methods on the Tecnick dataset.

[0027] Figure 8 Rate-distortion curves of different methods on the HEVC Class B test set.

[0028] Fig. 9 This is a comparison chart of the subjective visual quality of the method of the present invention, JPEG, BPG (4:4:4), JPEG2000 and comparative method 1 on the image data set Kodak. Figure (a) is the original image, Figure (b) is the image compressed by the present invention, Figure (c) is the image compressed by comparative method 1, Figure (d) is the image compressed by BPG (4:4:4), Figure (e) is the image compressed by JPEG2000, and Figure (f) is the image compressed by JPEG.

[0029] Fig.10 The figure is a comparison chart of the subjective visual quality of the method of the present invention, JPEG, BPG (4:4:4), JPEG2000 and comparative method 2 on the image data set Kodak. Figure (a) is the original image, Figure (b) is the image compressed by the present invention, Figure (c) is the image compressed by comparative method 2, Figure (d) is the image compressed by BPG (4:4:4), Figure (e) is the image compressed by JPEG2000, and Figure (f) is the image compressed by JPEG. DETAILED DESCRIPTION

[0030] The present invention is further described in detail below in conjunction with the embodiments. It is necessary to point out that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Technical personnel familiar with the relevant field may make some non-essential improvements and adjustments to the present invention according to the above invention content for specific implementation, which should still fall within the scope of protection of the present invention.

[0031] (1) The present invention proposes a full I-frame video compression method based on channel context. In order to verify the effectiveness of the proposed algorithm, the Kodak dataset, Tecnick dataset and HEVC standard test sequence, which are often used to test the image compression effect, are selected as experimental test sets. The mean of the compression results is taken as the final result, and the compression effect is measured. The metric used for the compression rate is the average bit rate per pixel (bits per pixel, bpp), which is calculated by dividing the total number of bits of the encoded image by the total number of pixels. The training set is the Vimeo90K dataset, which is a seven-frame continuous video sequence. The seven-frame training set is directly mixed and shuffled as a single image frame for input. The size of a single training image is 448×256, and the image is in RGB format. The image is randomly cropped to a size of 256×256 as the input of the neural network. The Adam optimizer is selected in the training stage. and The initial learning rate is set to 1× , after 50 epochs, it drops to 5× , the size of a single training group is 16, and the number of iterations of a single epoch of the training set is 40119.

[0032] (2) Input the training set into our network, train the five best models optimized by MSE and MS-SSIM, use the best models for the test set, and obtain the compressed images. We use MSE, MS-SSIM, and bpp to measure the compression effects of the method of the present invention and other methods.

[0033] (3) Figure 6 , 7 , 8 respectively give the rate-distortion curves of different methods on the Kodak dataset, Tecnick dataset and HEVC Class B test set. The present invention not only outperforms JPEG, JPEG2000, and BPG in the full bit rate segment, but also performs better than Hu2021 and Chen2021 in the low bit rate and medium bit rate segments. The rate-distortion curves of the video sequences of different test sets processed by the proposed full I compression network are all above all the compared methods, which shows that under the condition of the same bit rate, the reconstructed video PSNR of our proposed method is higher.

[0034] (4) From Fig. 9 , 10 It can be seen that the subjective effect of compressed video quality of our proposed method is better than that of other compression methods.

[0035] The comparison method is: Method 1: The method proposed by Y. Hu, W. Yang, Z. Ma et al., reference “Y. Hu, W. Yang,Z. Ma and J. Liu, "Learning End-to-End Lossy Image Compression: A Benchmark," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 8, pp. 4194-4211, 1 Aug. 2022, doi: 10.1109 / TPAMI.2021.3065339.”

[0036] Method 2: The method proposed by Lu M, Chen T, Liu H et al., reference “Fu H, Liang F.Learned image compression with generalized octave convolution and cross-resolution parameter estimation[J]. Signal Processing, 2023, 202: 108778.”.

Claims

1. A full I-frame video compression method based on channel context is characterized by Considering the correlation between potential feature channels and combining super prior information, the steps are as follows: (1) Input image First, it is sent to the main encoder network ( ), the input image x is converted into potential features y, combined with the potential features y obtained in the previous step and quantized and entropy encoded, and the quantized potential features are used Represented as follows, the encoder network consists of two transformation blocks: a residual downsampling layer and a residual enhancement layer. A lightweight residual block (LRB) is used to learn the features of the previous layer and combine them with the attention features. Based on the lightweight residual block, a pooling-based attention module is introduced in the middle of the encoding network to enhance the features of the non-local domain. At the end of the encoder, there is a downsampling convolution layer and a lightweight non-local residual attention layer different from the pooling-based attention to increase the receptive field. The compression process of the input image can be expressed by formula (1): in, is the parameter that needs to be optimized in the encoder network, and Q represents the quantization process; (2) The super prior model is introduced, and y passes through the super prior encoder ( Processing, the modules in the super prior network obtain implicit information and then transform it into new intermediate information , intermediate information Quantified as Then enter the arithmetic encoder to obtain the bit stream and the super prior decoding network ( Reconstruct the hyper-prior information and obtain the estimated parameters of the entropy model. The process is: in and yes and The corresponding optimization parameters are is the model estimate; (3) Recover the known probability distribution parameters and then Input decoder network ( ) to generate a reconstructed image, and the reconstructed frame is then passed through the quality enhancement module to output the image frame reconstructed under deep learning , the process is: in, is the parameter to be optimized in the decoder network, and E represents the quality enhancement module; (4) The present invention proposes to improve the performance of the entropy model based on a cross-channel context model: first, the latent features Divide into two groups of features by channel and , then the feature Get features through a mask convolution ,feature Generate features by searching the model across channels , the decoded features are combined with the super-prior output, and the masked 2D convolution and cross-channel search modules are used to fuse the super-prior and autoregressive prior for accurate entropy estimation.

Images