Video compression method research based on space-time entropy model and multi-reference time domain context network

Abstract

At present, most video compression technologies mainly depend on a predictive coding framework, and the method assumes that a current pixel and a pixel at the same position in a predictive frame have very high correlation, but neglects the relationship between adjacent pixels in the predictive frame, so that an entropy value is higher than an entropy value of conditional coding. Therefore, the invention provides a video compression method based on conditional coding, and the method optimizes the coding and decoding of the current video frame through an efficient space-time entropy model in combination with multi-reference time domain context information as conditional input, so that the advantages of conditional coding are brought into play to the greatest extent. Experimental results show that the method is superior to the existing video compression technology in the aspects of peak signal-to-noise ratio (PSNR) and multi-scale structural similarity (MS-SSIM).

Description

TECHNICAL FIELD

[0001] The present application will fully play the advantages of deep learning method, and proposes a video compression method based on space-time entropy model and multi-reference time domain context network. First, the advantages of efficient use of statistical redundancy and conditional entropy in the video compression method based on entropy model are combined to further improve the compression efficiency. Then, by fusing multiple video frames, multi-frame context information is generated to obtain more accurate context reference information. By multi-frame fusion, the temporal redundancy between frames is reduced, and more rich multi-frame temporal context information is generated. BACKGROUND

[0002] Currently, the research in the field of learning-based video compression can be roughly divided into four main directions, which are related and different. First, the residual coding-based scheme mainly focuses on reducing the redundancy information between video frames through pixel domain or feature domain prediction. Most of the research work adopts the auto-encoder coding structure, which represents the frames and the obtained residual in the latent space. Second, the 3D convolution-based video compression method regards the entire video sequence as a whole, and uses 3D convolution layers to analyze and compress the space-time information in the video. Unlike the residual-based method, the 3D convolution-based video coding pays more attention to the overall features of the entire video data rather than the differences between individual frames. Third, the entropy coding-based method regards video frames as separate pictures and processes each frame separately using image encoder and decoder, and analyzes the correlation of the latent representation between different frames to model the entropy and find the statistical dependence between frames to improve the coding efficiency. Finally, the conditional coding-based scheme uses the temporal context as the condition to automatically learn the correlation in the time dimension. The main difference between the conditional coding-based video compression method and the other three methods is its dynamic adaptability and the use of temporal relationship.

[0003] In existing entropy-based video compression techniques, compression quality can be improved by more accurately estimating the entropy distribution of latent variables generated by the encoder, with entropy modeling playing a crucial role. These entropy models typically employ a combination of a priori networks and context networks to optimize the encoder-decoder estimation of latent probabilities, enabling the utilization of statistical redundancy in video frames. However, this approach suffers from several major problems. First, the probability value of the current element to be encoded depends only on a small subset of already encoded elements, limiting the accuracy of long-term conditional probability estimation. Second, some entropy models employ autoregressive compression and decompression methods, requiring sequential computation, making parallel acceleration difficult. Furthermore, due to the limitations of shallow networks used in existing methods, the analysis and synthesis transformations fail to minimize spatial redundancy while maintaining reconstruction quality. Moreover, the information transmitted by the priori network is not fully and effectively utilized in the video frame reconstruction process. Finally, the interaction and cooperation mechanism between the priori network and the context model has not been fully studied and optimized. These issues affect the efficiency of video compression entropy models.

[0004] In conditional coding-based video compression, by focusing on the correlation between adjacent pixels in the predicted frame, the entropy of conditional coding is lower than the general information entropy of other entropy models. While the DCVC series of compression methods utilizes motion compensation and several convolutional layers to generate temporal contextual information from previously decoded single frames—a simple and useful approach—the contextual information generated from reference single frames loses much texture and motion information, and the contextual reference information generated from single frames is limited, failing to fully utilize the temporal correlation between video frames. Summary of the Invention

[0005] This invention fully leverages the advantages of deep learning methods. The video compression framework based on a spatiotemporal entropy model and multi-frame fusion context information mainly includes modules such as a motion estimation network, an MV encoder-decoder, a multi-reference temporal context network, a spatiotemporal entropy model network based on context and hierarchical super-prior networks, a context encoder-decoder, and a frame generator network. Detailed block diagrams are available in [link to block diagram]. Figure 1 This method mainly includes five steps, namely:

[0006] Step 1: Motion Estimation: Estimate the current frame x t and the previously reconstructed frame The motion between them, we will use the current frame x t And the previous frame that was previously decoded The input is fed into the motion estimation module of the neural network to estimate the optical flow, which represents the motion estimation vector for each pixel, denoted as v. t ;

[0007] Step 2: MV Encoder-Decoder: After obtaining the motion vector vt Then, we use an MV encoder and decoder to perform lossy compression and reconstruction on the input MV. Specifically, v t The MV, compressed and reconstructed by an autoencoder, is represented as follows:

[0008] Step 3: Multi-reference temporal context network: for motion vector buffer and information in the decoding buffer We use a multi-reference temporal context network to fuse features from multiple video frames and generate multi-frame context information. To obtain more accurate contextual information, while reducing temporal redundancy in the feature domain across multiple frames;

[0009]

[0010] Step 4: Spatiotemporal Entropy Model Based on Context and Hierarchical Super-Prior Networks: The entropy model is used for latent spatial estimation, rather than pixel-level spatiotemporal redundancy, which greatly reduces the complexity of the framework. We propose a spatiotemporal entropy model based on joint context and hierarchical super-prior networks for video compression, which directly compresses the original pixels. The entire entropy encoder and decoder consists of two parts: context and hierarchical super-prior parameter networks and joint entropy parameter encoder and decoder networks. The context and hierarchical super-prior parameter networks can not only reconstruct more accurate edge information using latent spatial information, but also establish the dependencies between pixels more accurately. The joint entropy parameter encoder and decoder networks can obtain more accurate joint entropy parameters, thereby effectively achieving compression.

[0011] Step 5: Context Encoder-Decoder and Frame Generator: Context information generated for multiple frames. With the help of a context encoder-decoder and a frame generator, the current frame is compressed and reconstructed. The calculation formula is as follows:

[0012]

[0013] The specific process is as follows:

[0014] (1) The video compression framework based on the spatiotemporal entropy model and multi-frame fusion context information designed in this invention mainly includes modules such as a motion estimation network, an MV encoder-decoder, a multi-reference temporal context network, a spatiotemporal entropy model network based on context and hierarchical advanced prior networks, a context encoder-decoder network, and a frame generator network. Detailed block diagrams are provided in the appendix. Figure 1 .

[0015] (2) Construction of the multi-reference temporal context network designed in this invention. Video consists of a series of consecutive frames, each a still image, with high similarity between adjacent frames. This similarity includes not only the almost unchanged background but also the generally coherent changes in the position and state of many dynamic objects. This temporal continuity allows us to predict or estimate the content of a frame based on information from previous frames. Considering previously decoded frames... With strong correlation to the current frame, we select video frames from some datasets to generate multi-frame context information visualizations. It is obtained by fusing the first four frames.

[0016] The proposed multi-reference temporal context network is shown in the attached figure. Figure 2 As shown in the right figure, since capturing the long-range pixel dependencies of global scene attributes has proven beneficial for a wide range of image restoration and image enhancement tasks, we leverage the Global Context Attention (GCA) mechanism to refine the latent representation generated by the residual blocks, as shown in the attached figure. Figure 3 As shown in the diagram on the right. Let x... b It is a potential representation of multiple channels of the input video frame. The computation process of our multi-reference temporal context network is as follows:

[0017]

[0018] in In the expression, W k This represents a convolutional layer with a filter of size k×k, and each W k Corresponding to individual layers with different parameters, γ represents Leaky ReLU activation, and ψ is softmax activation. This represents matrix multiplication, and α(·) represents global context attention.

[0019] After obtaining the merged frame, we perform a warp operation on the merged frame and the MV of the current frame and the previous four frames, as shown in the attached diagram. Figure 4 As shown, multi-frame context information is obtained.

[0020]

[0021] In acquiring context information containing multiple frames Subsequently, our entropy model will perform temporal and spatial processing on the context information of multiple frames to generate a representation containing global and local context information.

[0022] (3) The spatiotemporal entropy model of this invention based on context and hierarchical advanced prior networks is attached. Figure 5As shown, the context- and hierarchical hyperprior network-based spatiotemporal entropy model can be divided into context networks, hierarchical hyperprior networks, and encoder-decoder networks. Different context networks obtain other entropy parameters by finely capturing the correlation between different pixel positions in the video. Furthermore, the hierarchical hyperprior network consists of low-level and high-level hyperprior networks. The low-level hyperprior network acquires edge and texture information of video frames. The high-level hyperprior network aims to extract semantic and content information of video frames and assist the low-level hyperprior network in encoding and decoding. After acquiring the potential prior information of the video frames, the hierarchical hyperprior network processes this potential prior information to obtain high-level and low-level hyperprior parameters. These two types of parameters, containing contextual and edge information, jointly help the entropy model estimate video information.

[0023] In our entropy model, we first consider contextual information. With the current frame y t Perform global context information fusion, such as Figure 7 As shown. The expression is:

[0024]

[0025] in This represents the context modeling module, which uses a weighted average to combine the features of all pixel locations with their weights α. j The global context features are obtained by combining them; δ(·) represents the feature transformation to capture the dependencies between different channels; F(,,) represents the fusion function that aggregates the global context features with the features of each channel. represents the fusion function that aggregates the global context features with the features at each position.

[0026] As attached Figure 6 As shown, the fusion matrix With the potential representation y of the current frame t ψ is obtained by processing using a hierarchical hyperprior network. The calculation process is as follows:

[0027]

[0028] Where the parameter θ hd This represents the parameters of the hierarchical hyper-prior decoder.

[0029] In hierarchical hyperprior networks, the formula for calculating the analytic transformation of hierarchical hyperprior networks is as follows:

[0030]

[0031] in This represents the low-level analysis transformation. After obtaining the low-level prior information z1, we refine z1 using high-level super-prior information to obtain high-level super-prior information z2. Its calculation formula is:

[0032]

[0033] in This represents the low-level analysis transformation. The process for calculating the entropy parameter of the low-level advanced prior network, guided by the high-level advanced prior network, is as follows:

[0034]

[0035] Local context networks integrate the latent intrinsic relationships of local features, as shown in the appendix. Figure 7 As shown. It can be represented as:

[0036]

[0037] Where, θ cm For context parameters.

[0038] As attached Figure 8 As shown, in order to perform arithmetic encoding and decoding, the entropy encoder-decoder needs to estimate the probability of the symbols to be encoded from the latent representation quantized. The parameter estimation network computes the output of the context network. φ and the output ψ of the hierarchical super-prior decoder. The parameter estimation network takes the super-prior latent representation as input and samples a 5×5 region centered at the location to be predicted. Each sampled block is processed using a multi-layer convolutional sub-network to estimate the probability. In the last layer of the sub-network, the patch is flattened into a vector, and a fully connected layer is used to map the feature vector to the mean and scale vector of the current location. Local convolutions share the kernel along all spatial locations. Thus, the super-prior latent representation is densely sampled to complete the estimation without any latent information loss during the convolutional operations. The resulting multiple sets of entropy parameters are represented as (μ tk ,σ tk ), where k = 0, 1, 2, 3.

[0039]

[0040] We then combine the obtained entropy parameters. To more accurately estimate the probability distribution of the latent representation, we model the distribution of the quantized latent representation as a Laplace distribution:

[0041]

[0042] The advantages and beneficial technical effects of this invention compared with the prior art are as follows:

[0043] (1) Entropy model based on context network and hierarchical super-prior network. This method utilizes spatiotemporal characteristics and combines the principle of conditional coding to optimize the entropy coding of latent variables. By analyzing the latent representation correlations between different frames in the video sequence, the statistical dependencies between frames can be better utilized, thereby significantly improving the coding efficiency.

[0044] (2) Based on the multi-reference temporal context network, the multi-reference context information introduced by the network makes full use of the temporal correlation in the video coding process. By fusing multiple video frames, multi-frame context information is generated to obtain more accurate context reference information. Multi-frame fusion reduces inter-frame temporal redundancy information, generates richer multi-frame temporal context information, and improves compression efficiency.

[0045] (3) The learning-based end-to-end video coding method of this invention can effectively improve coding performance. Compared with HEVC (very-slow) mode, the proposed video coding method reduces BDBR by an average of 34.2% under the PSNR evaluation index and reduces BDBR by an average of 51.9% under the MS-SSIM evaluation index. Attached Figure Description

[0046] Figure 1 It is a video compression framework based on a spatiotemporal entropy model and multi-frame feature fusion context information.

[0047] Figure 2 For multi-reference temporal context networks.

[0048] Figure 3 For Warp operations.

[0049] Figure 4 This is a space-time entropy model based on context and hierarchical hyperprior networks.

[0050] Figure 5 For global context network.

[0051] Figure 6 It is a hierarchical super-prior network.

[0052] Figure 7 It is a local context network.

[0053] Figure 8 For parameter estimation network.

[0054] Figure 9 PSNR RD curves for testing the UVG, HEVC Class D, HEVC Class B, and MCL-JCV datasets.

[0055] Figure 10MS_SSIM RD curves for testing the UVG, HEVC Class D, HEVC Class B, and MCL-JCV datasets. Detailed Implementation

[0056] The present invention will be further described in detail below with reference to the embodiments. It should be noted that the following embodiments are only used to further illustrate the present invention and should not be construed as limiting the scope of protection of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to the present invention based on the above-described invention, and these improvements and adjustments should still fall within the scope of protection of the present invention.

[0057] (1) Datasets: Many datasets can be used to train learning-based video compression networks. Following most previous learning-based video compression schemes, we used video sequences from the Vimeo-90k dataset as our training data. During training, we randomly cropped the video frames of each sequence into 256x256 blocks to meet the model's requirements. To evaluate the performance of our proposed model in video compression, we used three datasets: UVG, MCL-JCV, and HEVC. These datasets contain various video scenes and are commonly used for performance evaluation of video compression algorithms. The UVG dataset contains 7 1080p sequences, and the MCL-JCV dataset contains 30 1080p sequences. The HEVC dataset contains 16 sequences. The source video sampling format of these datasets is YUV420. When calculating PSNR, we calculated it in the RGB color space and used the B and D classes of the HEVC dataset for testing.

[0058] (2) Implementation details: We trained different λ values ​​(λ = 256, 512, 1024, 2048) for different coding efficiencies. During training, the Adam

[47] optimizer was used and the batch size was set to 16. When using MS-SSIM for performance evaluation, we also adjusted the model by setting different λ values ​​(λ = 8, 16, 32, 64). Our model was implemented based on the PyTorch framework and trained on four NVIDIA 3090 GPUs. Training the model took one week.

[0059] (3) Evaluation Metrics: In the experiments, we used bits per pixel (Bpp) to measure the bit cost per pixel in each frame. To evaluate the distortion between the reconstructed frame and the original frame, we used Peak Signal-to-Noise Ratio (PSNR) and Multi-Scale Structural Similarity Index (MS-SSIM). Furthermore, we used BDBR to represent the average bit rate saving at the same reconstruction quality, and also calculated BD-PSNR. BD-PSNR represents the average gain of reconstruction quality at the same bit rate compared to the original frame.

[0060] (4) The invention first compares the proposed method with the HEVC standard based on the objective evaluation index PSNR. Table 2.2 shows the comparison of PSNR of the proposed method and the HEVC standard at different bitrates, as well as the comparison of BDBR results of different video sequences based on the PSNR index.

[0061] Method 1: The proposed method is compared with other methods, including the traditional method H.265 and recently proposed learning-based methods.

[0062] DVC, reference "Lu G, Ouyang W, Xu D, et al.Dvc: An end-to-end deep videocompression framework[C] / / Proceedings of the IEEE / CVF conference on computervision and pattern recognition.2019:11006-11015."

[0063] DVCPro, reference "Liu H, Shen H, Huang L, et al. Learned video compressionvia joint spatial-temporal correlation exploration[C] / / Proceedings of the AAAI Conference on Artificial Intelligence.2020,34(07):11580-11587."

[0064] NVC, reference "Liu H, Lu M, Ma Z, et al. Neural video coding using multiscale motion compensation and spatiotemporal context model [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 31(8): 3182-3196."

[0065] SSF, reference "Agustsson E, Minnen D, Johnston N, et al.Scale-space flow for end-to-end optimized video compression[C] / / Proceedings of the IEEE / CVFConference on ComputerVision andPattern Recognition.2020:8503-8512."

[0066] DCVC, reference "Li J, Li B, Lu Y.Deep contextual video compression[J].Advances in Neural Information Processing Systems, 2021,34:18114-18125."

[0067] DMVC, reference "Lin K, Jia C, Zhang

[0068] The method proposed by Zhang et al. is referenced in "Zhang X, Shao J, Zhang J. Low-complexity Deep Video Compression with A Distributed Coding Architecture[C] / / 2023IEEE International Conference on Multimedia and Expo(ICME).IEEE,2023:2537-2542.".

[0069] Method 2: Test results of UVG, HEVC Class D, HEVC Class B, and MCL-JCV datasets BDBR.

[0070] Table 1 shows the BDBR values ​​measured on the UVG, HEVC B and D classes, and MCL-JCV datasets under the PSNR metric.

[0071]

[0072]

[0073] Table 2 shows the BDBR values ​​measured on the UVG, HEVC B and D classes, and MCL-JCV datasets under the MS-SSIM metric.

[0074]

Claims

1. A research framework for video compression methods based on spatiotemporal entropy models and multi-reference temporal context networks, characterized by... Includes the following steps: Step 1: Motion Estimation: Estimate the current frame x t and the previously reconstructed frame The motion between them, we will use the current frame x t And the previous frame that was previously decoded The input is fed into the motion estimation module of the neural network to estimate the optical flow, which represents the motion estimation vector for each pixel, denoted as v. t ; Step 2: MV Encoder-Decoder: After obtaining the motion vector v t Then, we use an MV encoder and decoder to perform lossy compression and reconstruction on the input MV. Specifically, v t The MV, compressed and reconstructed by an autoencoder, is represented as follows: Step 3: Multi-reference temporal context network: for motion vector buffer and information in the decoding buffer We use a multi-reference temporal context network to fuse features from multiple video frames and generate multi-frame context information. To obtain more accurate contextual information while reducing temporal redundancy across multiple frames in the feature domain; Step 4: Spatiotemporal Entropy Model Based on Context and Hierarchical Super-Prior Networks: The entropy model is used for latent spatial estimation, rather than pixel-level spatiotemporal redundancy, which greatly reduces the complexity of the framework. We propose a spatiotemporal entropy model based on joint context and hierarchical super-prior networks for video compression, which directly compresses the original pixels. The entire entropy encoder and decoder consists of two parts: context and hierarchical super-prior parameter networks and joint entropy parameter encoder and decoder networks. The context and hierarchical super-prior parameter networks can not only reconstruct more accurate edge information using latent spatial information, but also establish the dependencies between pixels more accurately. The joint entropy parameter encoder and decoder networks can obtain more accurate joint entropy parameters, thereby effectively achieving compression. Step 5: Context Encoder-Decoder and Frame Generator: Context information generated for multiple frames. With the help of a context encoder-decoder and a frame generator, the current frame is compressed and reconstructed.

Images