Method and system for learning video compression

By enhancing the context mining model and transformer-based post-enhanced network, the problems of error propagation and high computing resource consumption in video compression are solved, and efficient video compression is achieved, which is better than traditional video encoding standards.

CN119182918APending Publication Date: 2024-12-24CITY UNIVERSITY OF HONG KONG

Patent Information

Application Number
CN202410422251.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-06-21
Filing Date
2024-04-09
Publication Date
2024-12-24

AI Technical Summary

Technical Problem

The existing video compression technology has error propagation problems and high computing resources when processing high-resolution videos. The block-based hybrid video compression framework is difficult to effectively utilize space and time redundancy, resulting in low compression efficiency.

Method used

A enhanced context mining model and a transformer-based backend network are proposed to reduce context channel redundancy through convolution and residual learning. The transposed gated converter block is designed to calculate the self-attention across channels, and fully resolved Rate pipelines to reduce information loss and calculation consumption.

Benefits of technology

It effectively reduces error propagation, improves compression efficiency, can process high-resolution frames on a single GPU, and is superior to traditional video encoding standards in PSNR and MS-SSIM indicators, saving bit rate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119182918A_ABST
    Figure CN119182918A_ABST
Patent Text Reader

Abstract

There is provided a computer-implemented method for learning video compression, the method comprising processing a current frame (xt) and a previously decoded frame # imgabs0 # of video data using a motion estimation model to estimate a motion vector (vt) for each pixel, compressing the motion vector (vt) and reconstructing the motion vector (vt) into a reconstructed motion vector # imgabs1 # applying an enhanced context mining (ECM) model, an enhanced context # imgabs4 # is obtained from a reconstructed motion vector # imgabs2 # and a previously decoded frame feature # imgabs3 #, and a current frame (xt) is compressed with the help of an enhanced context # imgabs5 # to obtain a reconstructed frame # imgabs6 # and provide the reconstructed frame # imgabs7 # to an enhanced back-end network to obtain a high-resolution frame # imgabs8 #, compared with previous traditional video codecs H.264 and H.265, the DEVC of the invention realizes a significant performance gap, and has the advantages of high resolution and high resolution. This shows a huge potential of a learning-based method.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention generally relates to methods and systems for learning (learning-based) video compression. Background Art

[0002] In the past few decades, video coding standards such as Advanced Video Coding (AVC) [1], High Efficiency Video Coding (HEVC) [2], and Versatile Video Coding (VVC) [3] have all followed the classic block-based hybrid video compression framework and various designs have been developed. However, with the rapid development of video applications and the launch of ultra-high-definition video services, the growth of video data has exceeded the scope of compression ratio improvement [4]. Therefore, it is particularly important to explore video compression techniques, for example, how to provide high-quality video data at a given bit rate when network and storage capabilities are limited. With the application and success of deep learning due to its powerful representation ability [5]-[7], learning-based compression techniques have attracted people's interest and achieved improvements, so they can be further explored [8]-

[10] .

[0003] For learning-based image compression, Balle et al. proposed a basic CNN (convolutional neural network) framework in

[11] . They transformed the image into a latent code and then reconstructed the image using the inverse transform. To make the entire framework trainable end-to-end, uniform noise was added during training for quantization. By using a single loss function for full joint optimization training, learning-based image compression using discrete Gaussian mixture likelihood

[12] achieved comparable performance to the latest traditional video standard VVC, revealing the effectiveness of neural networks in eliminating spatial redundancy.

[0004] In addition to spatial redundancy, temporal redundancy is also a part of video context that needs to be reduced. Learning-based video compression technology has also made progress. Lu et al. proposed a predictive end-to-end learning video compression framework (DVC) in

[13] . In DVC, the residual between the input frame and the predicted frame is calculated. Then the motion vector and the residual are compressed separately using an entropy model. However, the entropy of residual coding is greater than or equal to the entropy of conditional coding

[14] . Therefore, Li et al. proposed a deep contextual video compression framework (DCVC) to extract valuable context as the condition for the context encoder-decoder and entropy model to compress the current frame

[15] . Although DCVC has been developed and has shown the effectiveness of the conditional coding paradigm with the release of test code, improvements to DCVC may be desirable. Summary of the invention

[0005] In DCVC, there is only one residual block (Resblock) and one convolutional layer for context refinement. Moreover, the context is generated without supervision and there may be redundancy between context channels. However, the context is very important in the conditional framework because it will be introduced into the context encoder-decoder and entropy model for compression.

[0006] Therefore, some embodiments of the present invention propose an enhanced context mining (ECM) model to reduce the redundancy between context channels. Specifically, in order to utilize high-dimensional context, some embodiments of the present invention propose convolution and residual learning and context channels. Therefore, for context and entropy coding models, the potential clean context in the hidden layer is implicitly retained.

[0007] In addition, there is also the problem of error propagation in DCVC. Figure 1a-1b As shown, the reconstructed error will propagate to the next frame, resulting in a decrease in frame quality and an increase in bit consumption. At the same time, the in-loop filtering technology in some video codecs can further improve the quality of the reference frame, thereby improving the compression efficiency

[16] -

[18] . Inspired by the successful in-loop filtering technology in some video codecs, some embodiments of the present invention utilize a post-enhancement back-end network to perform learning-based video compression and alleviate the error propagation problem in GOP (group of pictures). In addition, considering that the transformer has shown good performance in processing natural language tasks

[19] ,

[20] and visual problems

[21] -

[23] , some embodiments of the present invention provide a transformer-based post-enhancement back-end network to improve compression efficiency. The challenge of transformer adaptation to video comes from the high resolution of pixels and the huge variation in the scale of visual entities, which leads to a lot of computation and GPU memory consumption. Therefore, some embodiments of the present invention provide a transposed gated transformer block for calculating self-attention across channels rather than spatial dimensions. The size of the attention map can be Rather than Considering that down / up sampling will lead to information loss and inevitably introduce artifacts, some embodiments of the present invention provide a full-resolution pipeline. At the same time, to prevent performance degradation, a transformer block without layer normalization is designed (the benefits of canceling layer normalization are demonstrated in an ablation study). Some embodiments of the present invention design the transformer structure as the inner loop of a learning-based video compression task, which is feasible for high-resolution frames using a single GPU.

[0008] In some embodiments of the present invention, by enhancing the context mining model and the transformer-based post-enhancement back-end network, less error propagation and better compression efficiency can be obtained. At the same time, the model proposed in some embodiments can be extended to other learning coding methods extended from the DCVC framework. The contributions of some embodiments of the present invention include:

[0009] (i) An enhanced context mining (ECM) model is proposed to reduce the redundancy across context channels. The latent clean context is obtained through convolution and residual learning. The enhanced context is introduced into the context encoder-decoder and entropy model.

[0010] (ii) A transformer-based post-enhancement backend network is included in the deep video compression network to further alleviate the error propagation problem. In particular, the backend network adopts a full-resolution pipeline because down / up sampling will introduce artifacts and information loss. In addition, a transposed gated transformer block without layer normalization is designed, and the self-attention of each channel (rather than the spatial dimension) is calculated to reduce the computation and GPU memory consumption, so that the entire framework can achieve high-resolution frames with a single GPU.

[0011] (iii) A dual-enhancement video compression framework is provided using an enhanced context mining model and a transformer-based post-enhancement network. The entire network is jointly optimized to exploit both spatial and temporal information. Experiments and analysis validate the effectiveness of the proposed model in one implementation, which achieves superior performance in MS-

[0012] In terms of the SSIM indicator, 6.7% of bits are saved. In terms of the PSNR indicator, the model in one embodiment outperforms HEVC, with an average bit saving of 36.40%, a UVG sequence saving of up to 46.33%, and a HEVCB-type sequence saving of 41.06%.

[0013] According to one aspect of the present invention, there is provided a computer-implemented method for learning video compression, the method comprising processing a current frame (x) of video data using a motion estimation model t ) and previously decoded frames To estimate the motion vector (v) of each pixel t ), compressed motion vector (v t ) and the motion vector (v t ) is reconstructed into a reconstructed motion vector The Enhanced Context Mining (ECM) model is applied to reconstruct motion vectors and previously decoded frame features Get enhanced context In the enhanced context Compress the current frame with the help of t ) to obtain the reconstructed frame and will reconstruct the frame Provided to the enhanced backend network to obtain high-resolution frames

[0014] In some implementations, the motion estimation model may be based on a spatial pyramid network.

[0015] In some implementations, applying an enhanced context mining (ECM) model may include utilizing cross-channel interactions and residual learning operations to reduce redundancy across context channels.

[0016] In some embodiments, applying an enhanced context mining (ECM) model may include: t ) and previously decoded frames Get motion vector and previously decoded frame features Motion vector and decoded frame features Perform warpage processing to obtain warpage characteristics And use residual blocks and convolutional layers to process warping features To get the context

[0017] In some embodiments, the application of the enhanced context mining (ECM) model may further include in the context Combine ReLU (rectified linear unit) and convolution to obtain residual context (R C ), and the residual context (R C ) added to the context Get enhanced context

[0018] In some embodiments, convolution is combined with ReLU to context The above may include context Through multiple layers of convolution and ReLU, and then through another convolution layer.

[0019] In some implementations, the enhanced context mining (ECM) model may be designed without using batch normalization layers.

[0020] In some implementations, compressing the current frame (x t ) may include converting the current frame (x t ) and enhanced context Connect together and process the current frame (x t ) and enhanced context To obtain the latent code (y t ), and in the enhanced context The latent code is converted back to pixel space with the help of

[0021] In some implementations, the post-enhancement backend network may be transformer based.

[0022] In some implementations, the post-enhancement back-end network may include a plurality of transposed gated transformer blocks (TGTBs) and a plurality of convolutional layers.

[0023] In some embodiments, the reconstructed frame Providing to the post-enhancement backend network may include applying a convolutional layer to the reconstructed frame To obtain low-level feature embeddings Where H×W is the spatial height and width, C represents the channel number; one or more transformer blocks are used to process the low-level features (F0) to obtain refined features (F R ), and apply convolutional layers to refine the features (F R ) to obtain the residual image Will rebuild the frame Add to the residual image to obtain

[0024] In some embodiments, at least one transformer block may include a transposed gated transformer block (TGTB) whose design does not require layer normalization.

[0025] In some embodiments, at least one transformer block may be modified to include multi-head transposition attention (MTA) and a gated feedforward network (GFN).

[0026] In some implementations, multi-head transposed attention (MTA) may include computing self-attention across channels.

[0027] In some implementations, multi-head transposed attention (MTA) may include applying depthwise convolutions.

[0028] In some embodiments, multi-head transposition attention (MTA) can generate a query (Q), a key (K), and a value (V) projection with local context from the feature input, and reshape the query (Q) into Reshape the key (K) to To obtain the size The transposed attention map of .

[0029] In some embodiments, a gated feedforward network (GFN) may include a gating mechanism and a depthwise convolution, wherein the gating mechanism may be implemented by elementwise product of parallel paths of two transformation layers, wherein one transformation layer is activated by a GELU nonlinearity, and the depthwise convolution may be applied to obtain information from spatially adjacent pixel locations.

[0030] In another aspect of the present invention, a system for learning video compression is provided, the system comprising one or more processors and a memory storing one or more programs, the programs being configured to be executed by the one or more processors. The one or more programs include instructions for performing or facilitating the performance of the above-mentioned computer-implemented method.

[0031] In some embodiments, a system for learning video compression is provided. The system includes a motion estimation model configured to receive a current frame (x) of video data. t ) and previously decoded frames To estimate the motion vector (v) of each pixel t ); a motion vector (MV) encoder and decoder configured to compress a motion vector (v t ) and the motion vector (v t ) is reconstructed into a reconstructed motion vector Enhanced Context Mining (ECM) model, which is configured to reconstruct motion vectors from and previously decoded frame features Get enhanced context A context encoder and a decoder configured to enhance the context Compress the current frame with the help of t ) to obtain the reconstructed frame and a post-enhancement backend network configured to reconstruct frames based on Get high-resolution frames

[0032] In some implementations, the motion estimation model may be based on a spatial pyramid network.

[0033] In some implementations, an enhanced context mining (ECM) model may be configured to utilize cross-channel interactions and residual learning operations to reduce redundancy across context channels.

[0034] In some embodiments, the enhanced context mining (ECM) model can be configured to be based on the current frame (x t ) and previously decoded frames Get motion vector and previously decoded frame features Motion vector and decoded frame features Perform warpage processing to obtain warpage characteristics And use residual blocks and convolutional layers to process warping features To get the context

[0035] In some embodiments, the enhanced context mining (ECM) model can be further configured as a context Combine ReLU and convolution to obtain residual context (R C ), and the residual context (R C ) added to the context Get enhanced context

[0036] In some embodiments, the enhanced context mining (ECM) model can be configured to Through multiple layers of convolution and ReLU, and then through another convolution layer, the convolution and ReLU are combined into the context

[0037] In some implementations, the enhanced context mining (ECM) model may be designed without using batch normalization layers.

[0038] In some embodiments, the context encoder and decoder may be configured to pass the input frame (x t ) and enhanced context Connect together to compress the current frame (x t ), the input frame (x t ) and enhanced context Input the context encoder to obtain the latent code (y t ), and in the enhanced context The latent code is converted back to pixel space with the help of

[0039] In some implementations, the post-enhancement backend network may be transformer based.

[0040] In some embodiments, the post-enhancement back-end network may include a plurality of transposed gated transformer blocks (TGTBs) and a plurality of convolutional layers.

[0041] In some embodiments, the post-enhancement backend network may be configured to reconstruct the frame Apply convolutional layers to obtain low-level feature embeddings To get high resolution frames Where H×W is the spatial height and width, C represents the channel number, and the low-level features (F0) are fed through one or more transformer blocks to obtain refined features (F R ), and refine the features (F R ) Apply convolutional layers to obtain the residual image Will rebuild the frame Add to this residual image to obtain

[0042] In some embodiments, at least one transformer block may include a transposed gated transformer block (TGTB) whose design does not require layer normalization.

[0043] In some embodiments, at least one transformer block may be modified to include multi-head transposed attention (MTA) and a gated feedforward network (GFN).

[0044] In some embodiments, multi-head transposed attention (MTA) can be configured to compute self-attention across channels.

[0045] In some embodiments, multi-head transposed attention (MTA) can be configured to apply deep convolution.

[0046] In some embodiments, the multi-head transposed attention (MTA) may be configured to generate a query (Q), a key (K), and a value (V) projection with local context from the feature input, and reshape the query (Q) into Reshape the key (K) to To obtain the size The transposed attention map of .

[0047] In some embodiments, a gated feedforward network (GFN) may include a gating mechanism and a depthwise convolution, where the gating mechanism can be implemented as an element-wise product of parallel paths of two transformation layers, one of which is activated by a GELU nonlinearity, and the depthwise convolution can be used to obtain information from spatially adjacent pixel locations.

[0048] In another aspect of the present invention, a non-transitory computer-readable medium is provided, on which instructions are stored, which, when executed by one or more processors, can cause the one or more processors to perform the above-mentioned computer-implemented method.

[0049] Other features and aspects of the invention will become apparent from consideration of the detailed description and accompanying drawings.Any feature described herein in relation to one aspect or embodiment may be appropriately combined with any other feature described herein in relation to any other aspect or embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings:

[0051] Figure 1a An example of peak signal-to-noise ratio (PSNR) comparison between DCVC and an embodiment of the present invention (GOP=10) is shown.

[0052] Figure 1b An example of bit cost comparison between DCVC and an embodiment of the present invention (GOP=10) is shown.

[0053] Figure 2A schematic diagram of an overall framework of dual enhancement modeling for learning video compression according to an embodiment of the present invention is shown.

[0054] Figure 3 An example architecture of an enhanced context mining model according to one embodiment of the present invention is shown.

[0055] Figure 4 An example architecture of a context encoder and decoder according to one embodiment of the present invention is shown.

[0056] Figure 5 An example architecture of a contextual temporal encoder according to one embodiment of the present invention is shown.

[0057] Figure 6 An example of an entropy model structure according to an embodiment of the present invention is shown.

[0058] Figure 7 The context according to one embodiment of the present invention is shown and enhanced context A visual example of .

[0059] Figure 8 An example of a transformer-based post-enhancement backend network architecture according to one embodiment of the present invention is shown.

[0060] Fig. 9 An example architecture of multi-head transposed attention according to one embodiment of the present invention is shown.

[0061] Fig.10 An example architecture of a gated feed-forward network according to one embodiment of the present invention is shown.

[0062] Fig.11a The figure shows the comparison of the encoding performance of the embodiment of the present invention and the existing codec when using the HEVC Class B dataset evaluated according to the PSNR indicator.

[0063] Fig.11b The figure shows the comparison of the encoding performance of the embodiment of the present invention and the existing codec when using the HEVC Class C data set evaluated according to the PSNR indicator.

[0064] Fig.11c The figure shows the comparison of the encoding performance of the embodiment of the present invention and the existing codec when using the HEVC Class D dataset evaluated according to the PSNR indicator.

[0065] Fig.11d The figure shows the comparison of the encoding performance of the embodiment of the present invention and the existing codec when using the HEVC Class E dataset evaluated according to the PSNR indicator.

[0066] Fig.11e The figure shows the comparison of the encoding performance of the embodiment of the present invention and the existing codec when using the UVG dataset evaluated according to the PSNR indicator.

[0067] Fig.11f The figure shows the coding performance comparison between the embodiment of the present invention and the existing codec when using the MCL-JCV dataset evaluated according to the PSNR indicator.

[0068] Fig.12a The figure shows the comparison of the encoding performance of the embodiment of the present invention and the existing codec when using the HEVC Class B dataset evaluated according to the MS-SSIM indicator.

[0069] Figure 12b The figure shows the comparison of the encoding performance of the embodiment of the present invention and the existing codec when using the HEVC Class C data set evaluated according to the MS-SSIM indicator.

[0070] Fig.12c The figure shows the comparison of the encoding performance of the embodiment of the present invention and the existing codec when using the HEVC Class D dataset evaluated according to the MS-SSIM indicator.

[0071] Fig.12d The figure shows the comparison of the encoding performance of the embodiment of the present invention and the existing codec when using the HEVC Class E dataset evaluated according to the MS-SSIM indicator.

[0072] Fig.12e The figure shows the coding performance comparison between the embodiment of the present invention and the existing codec when using the UVG dataset evaluated according to the MS-SSIM indicator.

[0073] Fig.12f The figure shows the coding performance comparison between the embodiment of the present invention and the existing codec when using the MCL-JCV dataset evaluated according to the MS-SSIM indicator.

[0074] Fig.13 Shows subjective quality comparison of VVC, DEVC embodiment and DCVC.

[0075] FIG. 14A shows a PSNR comparison between DCVC and DEVC embodiments (λ=2048).

[0076] FIG. 14B shows a bit rate comparison between DCVC and DEVC embodiments (λ=2048).

[0077] FIG. 14C shows a PSNR comparison between DCVC and DEVC embodiments (λ=1024).

[0078] FIG. 14D shows a bit rate comparison between DCVC and DEVC embodiments (λ=1024).

[0079] FIG. 14E shows a PSNR comparison between DCVC and DEVC embodiments (λ=512).

[0080] FIG. 14F shows a bit rate comparison between DCVC and DEVC embodiments (λ=512).

[0081] FIG. 14G shows a PSNR comparison between DCVC and DEVC embodiments (λ=256).

[0082] FIG. 14H shows a bit rate comparison between DCVC and DEVC embodiments (λ=256).

[0083] Fig.15 An example of an information handling system is shown in which some embodiments of the present invention may be used.

[0084] Before explaining any embodiments of the present invention in detail, it should be understood that the application of the present invention is not limited to the implementation details and component arrangements described in the following description or shown in the following figures. The present invention can have other embodiments and can be implemented or executed in various ways. In addition, it should be understood that the words and terms used herein are for descriptive purposes only and should not be regarded as limiting. DETAILED DESCRIPTION

[0085] Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0086] The deep contextual video compression framework (DCVC) with conditional coding paradigm extracts context and uses it as a condition for context encoder-decoders and entropy models. However, the generation of key context remains challenging. Some embodiments of the present invention propose an enhanced context mining model to reduce redundancy across context channels, where cross-channel interactions and residual learning are exploited to obtain better context features. In addition, inspired by the successful in-loop filtering in traditional codecs, some embodiments of the present invention provide a transformer-based post-enhancement network designed to alleviate the error propagation problem. Specifically, some embodiments of the present invention propose a full-resolution pipeline for information-free in-loop filtering without down / up sampling. In addition, in order to reduce the large amount of computation and memory consumption of the transformer, some embodiments of the present invention propose a transposed gated transformer block to calculate self-attention across channels rather than spatial dimensions. With the above design, the entire framework is feasible for high-resolution frames and can be jointly optimized to exploit spatial and temporal information.

[0087] Related Work

[0088] A. Learning Image Compression

[0089] Image compression based on deep learning has been a hot topic for end-to-end training. Instead of manually designing components based on experience, such as JPEG

[24] , JPEG2000

[25] , and BPG

[26] , learning-based image compression techniques map images to latent codes through transformations based on nonlinear networks. The latent codes are then quantized and written into the bitstream. Subsequently, the image can be reconstructed from the latent codes through inverse transformations. Reference

[11] proposed a whole trainable framework that can jointly optimize the estimated bit cost and the quality of the reconstructed image. Specifically, Balle′ et al. added uniform noise as a soft approximation to the vectorized latent codes during training and estimated the bit cost using a factorized entropy model. Subsequently, Balle′ et al. further proposed in

[27] to use a hyper-prior associated with side information to reduce the spatial redundancy of the latent codes and modeled the latent codes as zero-mean Gaussians with their own standard deviations. In addition,

[28] ,

[29] also incorporated autoregressive priors into the entropy model using masked convolutional networks. At the same time, Fabian Mentzer et al. proposed a conditional probability model for learning latent codes using 3D-CNN in their paper

[30] , while Hu et al. proposed a coarse-to-fine entropy model to utilize the hyper-prior of different layers

[31] . In addition, Cheng et al. proposed a flexible Gaussian Mixture Likelihoods to parameterize the distribution of latent codes in

[12] , achieving performance comparable to Versatile Video Coding (VVC). In addition to the CNN-based framework

[32] , recurrent neural network-based methods

[33] -

[35] and GAN-based frameworks

[36] ,

[37] have also been proposed for learning image compression.

[0090] B. Learning Video Compression

[0091] In addition to image compression, learning-based video compression has also developed rapidly in the past few years

[38] ,

[39] . Lu et al. first proposed a low-latency predictive end-to-end video compression framework (DVC) in

[13] ,

[40] , where all components are implemented by neural networks and optimized through a single rate-distortion trade-off loss. At this time, optical flow replaces the typical MV to estimate motion, while motion compensation relies on the reconstructed MV. The MV and residual between the predicted value and the ground truth are then encoded separately through image compression methods. Based on DVC, Hu et al. further proposed adaptive stream coding and introduced multi-resolution representation at block and frame levels in

[41] . Subsequently, Hu et al. proposed a feature space video compression framework (FVC), which greatly improved the compression performance. In FVC, motion estimation, compensation, compression, and residual compression are all performed in feature space rather than pixel space

[42] . Subsequently,

[43] proposed a coarse-to-fine deep video compression framework based on multi-resolution and feature space operations to achieve better compensation. To better handle motion, Agustsson et al. proposed a scale-space flow for learning video compression

[44] . In addition, Lin et al. proposed the Decomposed Motion Paradigm (DMVC) for Learning Video Compression in

[45] . Reference

[46] proposed enhanced motion compensation to generate better predicted frames. In addition, Fabian et al. proposed a video compression transformer (VCT) to directly learn the relationship between frames

[47] .

[0092] In addition to the above methods, it is worth mentioning that Li et al. transformed the predictive coding paradigm into the conditional coding paradigm and proposed the deep contextual video compression framework (DCVC) in

[15] . Subsequently, Sheng et al. proposed a method to extract multi-scale temporal context and maintain propagation features based on the conditional coding paradigm (DCVC)

[48] . In addition, based on conditional coding, Li et al. proposed a hybrid space-time entropy model for learning video compression in

[49] combined with the contribution of

[48] . Different from

[46] ,

[48] , and

[49] , the embodiments of the present invention focus on generating better context using the conditional coding paradigm

[15] .

[0093] Not limited to the previous frame as a reference, Lin et al. proposed multi-frame prediction to generate multiple MV fields in

[50] . Yang et al. introduced recurrent neural networks into video compression in

[51] and proposed a hierarchical learning video compression (HLVC) framework with three hierarchical quality layers and a recursive enhancement network in

[52] . Reza et al. proposed to extend the P-frame codec to B-frame coding using a frame interpolation method in

[53] .

[0094] C. Loop Filter

[0095] The deblocking filter and the sample adaptive offset filter

[54] are two types of in-loop filters specified in HEVC [2] and VVC [3]. They are applied after inverse quantization and before the frame is saved to the decoded frame buffer. In particular, the deblocking filter is designed to weaken the discontinuities at the boundaries of prediction and transform blocks. Sample adaptive offset improves the frame quality after the deblocking filter by reducing ringing artifacts. Using these two in-loop filters, better quality reference frames can be obtained, thereby improving compression efficiency.

[0096] With the development of deep learning, researchers have explored deep learning-based in-loop filtering enhancement techniques in HEVC and VVC. Da et al.

[55] proposed a residual learning CNN (VRCNN) with variable filter size to improve compression performance. In addition, residual highway units

[56] , switchable deep learning methods

[57] , context-aware CNNs

[58] , and enhanced deep convolutional neural networks

[59] for HEVC in-loop filtering have been proposed. Subsequently, Pham et al. proposed a learning-based spatiotemporal in-loop filtering method to improve the default in-loop filtering of VVC by taking advantage of the coding information

[16] . Zhang et al. proposed another specific CNN to enhance the random access (RA) mode in VVC

[17] . In addition, Ma et al. designed an MFRNet for post-processing and in-loop filtering of traditional video compression

[18] .

[0097] Although in-loop filtering technology in traditional video compression has been studied and used, in-loop enhancement technology for end-to-end video compression still needs to be further explored. The embodiment of the present invention designs a post-enhancement backend network to further improve the efficiency of end-to-end video compression.

[0098] method

[0099] A. Overall Framework

[0100] Figure 2 The overall framework of dual enhancement modeling for learning video compression in one embodiment of the present invention is shown. Let x t Represents the current frame, and x is estimated by motion estimation t With previously decoded frame The motion vector (MV) v between t , and then the MV is encoded and reconstructed through the MV encoder and decoder into By reconstructing the motion vector and previously decoded frame features Warping can get context This paper proposes Enhanced Context Mining (ECM) to learn enhanced context Then Input context encoder-decoder and entropy model to compress x t and refactored to The enhanced backend network is designed to further improve Become a frame Stored in the frame buffer for future reference. t ,y t is a potential representation, is the corresponding quantized version. The encoder uses all modules, while the decoder only includes modules with light gray boundaries. The entire framework is optimized using the rate-distortion (RD) loss. Figure 2 The overall framework of .

[0101] The overall framework of the proposed method is as follows Figure 2 As shown, by previously decoding the frame Get feature context And is the condition for the current frame x t Encode and decode to obtain the reconstructed frame It can be expressed as:

[0102]

[0103] f context (*) is the context generation function. f enc (*) and f dec (*) indicates context encoder and decoder. The flow of the whole framework is as follows.

[0104] First, the current frame x t and previously decoded frames The motion estimation model is fed into the estimation of the optical flow, which is considered as the estimated motion vector v for each pixel. t . Get the motion vector v t After that, use the MV encoder-decoder to t Next, we will apply enhanced context mining technology to obtain the reconstructed motion vector. and previously decoded frame features Learning context Then the context will be enhanced Refill the context encoder-decoder and entropy model to compress the current frame x t After the context decoder, the reconstructed frame is obtained. Considering that in-loop filtering can further improve compression efficiency, is fed into a transformer-based post-enhancement backend network to generate Then, is stored and transmitted for the next frame compression.

[0105] According to some embodiments, the motion estimation model is based on a spatial pyramid network (Spynet)

[60] . In addition, some embodiments adopt the DCVC

[15] method to refill the context into the context encoder-decoder and the entropy model.

[0106] Figure 2 The main components of the framework shown are:

[0107] Motion estimation: The current frame (x t ) and previously decoded frames The motion estimation model is fed into the estimation of the optical flow, thereby exploiting the temporal relationship. The optical flow is considered as the estimated motion vector v for each pixel. t According to one embodiment, the motion estimation model is based on a spatial pyramid network (Spynet).

[0108] MV encoder-decoder: get motion vector v t Then, the motion vector v is decoded using the MV encoder-decoder t Compress and restructure. is the reconstructed motion vector.

[0109] Enhanced context mining: Enhanced context mining is to reconstruct motion vectors and previously decoded frame features Learn richer context Then the context will be enhanced Refill into the context encoder-decoder and entropy model to improve compression efficiency. The details of the enhanced context mining model will be further explained in item B below.

[0110] Context Encoder-Decoder: In the context of enhanced With the help of the context encoder and decoder, the current frame x is compressed. t According to one embodiment, DCVC can be used to convert context Concatenate to frame x t , and then input to the context encoder-decoder.

[0111] Post-enhancement backend network: After passing through the context decoder, the reconstructed frame is obtained Considering that in-loop filtering can further improve the compression efficiency, this paper proposes a transformer-based post-enhancement back-end network to further improve the compression efficiency, which will be further explained in Section C below.

[0112] Entropy model: In the entropy model, the hierarchical prior and the spatial prior as well as the contextual temporal prior are fused with the Laplace distribution to build a contextual latent code model. In addition, there is also a corresponding entropy model for the MV latent code. However, according to one embodiment, only the spatial preamble and the lookahead preamble are applicable to the MV latent code. In addition, the arithmetic encoder can be used to write and read the bitstream.

[0113] The detailed information of the enhanced context mining model and the transformer-based post-enhancement backend network according to the embodiment of the present invention will be further introduced below.

[0114] B. Enhanced context mining

[0115] From the whole framework, we can see that context is crucial because it helps the context encoder-decoder and entropy model to compress the current frame. However, the motion estimation information obtained by optical flow is not accurate

[61] and may be and The bilinear warping stage introduces artifacts. In order to fully utilize the high-dimensional context features, it is proposed to reduce the redundancy between context channels through convolution operations and residual learning. An implementation of the proposed enhanced context mining model is shown in Figure 3 As shown. The numbers represent the channel dimensions. t and previously decoded frames Motion Vector and previously decoded frame features is used to generate context features, where the number of channels of frame features is set to 64 in

[15] . and The number of channels is , and bilinear interpolation is used as the warping operation to obtain the warping features It contains both motion information and the features of the previous frame. Then the warping features Input ResBlock and convolution layer to get context In DCVC

[15] , the context is refilled into the context encoder-decoder and entropy model as a compression condition. is generated without supervision, the context There may be redundancy between channels. Therefore, the present invention proposes to reduce redundancy through cross-channel interaction and residual learning operations. By combining convolution with ReLU, the network can gradually separate the context structure from the redundant observations through the hidden layer and obtain the residual context R C Finally, the residual context R C Join Get enhanced context The whole process of ECM is as follows.

[0116]

[0117]

[0118]

[0119] in, represents the convolutional layer, Indicates convolution plus ReLu layer. Figure 3 Six layers of convolution + ReLU are set for part A, followed by another convolution layer. In addition, Wu et al.

[62] pointed out that batch normalization works well when the batch size is greater than 16. When the batch size is less than 16, batch normalization will degrade the performance. Due to the limitation of GPU memory for high-resolution videos, the training batch size is set to 4 or 8 in previous learning-based video compression works

[42] -

[44] . To prevent performance degradation, the above-mentioned enhanced context mining model is designed without batch normalization layers.

[0120] Get enhanced context After that, enhance the context It is re-filled into a learning video compression framework, which includes a context encoder-decoder and an entropy model. Enhancing context features improves compression efficiency.

[0121] Figure 4 The details of refilling the context into the context encoder and decoder in one embodiment are shown. GDN means generalized division normalization, while IGDN is inverse GDN. The numbers are channel sizes. First, the input frame x t and enhanced context Then they are input into the context encoder to obtain the latent code y of the entropy model. t The context decoder is symmetrical with the encoder and uses the enhanced context Convert the latent code back to pixel space. This gives us the reconstructed frame. Figure 5 As shown, in order to reduce the temporal correlation between latent encoders, the context is enhanced is used to generate the temporal prior. Figure 5 The numbers in are the channel sizes. In addition, Figure 6 The entropy model structure in one embodiment is shown. t is the super prior, S t is the spatial prior, T t is the temporal prior. Q represents quantization. AE and AD are the arithmetic encoder and decoder, respectively.

[0122] Some embodiments of the present invention follow DCVC

[15] to assume that the underlying y tThe distribution of follows the Laplace distribution. The time prior T t With the hyper-prior H t

[27] and the spatial prior S t

[28] are combined to estimate the potential The mean and variance of the bitstream are obtained by using an arithmetic encoder. In addition, an arithmetic encoder is used to write and read the bitstream. At the same time, the MV latent coding also has a corresponding entropy model. According to an embodiment of DCVC, spatial prior and super prior are applied to MV latent coding.

[0123] Figure 7 It is context and enhanced context Visualization of the first four corresponding channels of . Their difference Δ is based on and In addition, Figure 7 It also calculates and displays and These differences have been eliminated by the enhanced context model mentioned above.

[0124] C. Post-enhancement backend network

[0125] Inspired by the successful application of transformers in natural language

[19] ,

[20] and vision problems

[21] –

[23] , in order to capture long-range pixel interactions to further improve performance, this paper proposes an enhanced background network, which is a transformer-based network. In the transformer, the complexity of self-attention grows quadratically with the spatial resolution of the input frame. Taking an H×W pixel image as an example, the complexity of the key-query dot product interaction is Therefore, some embodiments of the present invention propose a Transposed Gated Transformer Block (TGTB) to capture long-range pixel interactions and make the transformer and the entire framework applicable to high-resolution frames simultaneously. Figure 8 The post-enhancement back-end network proposed in one embodiment is shown. In this embodiment, the post-enhancement back-end network includes six transposed gated transformer blocks and two convolutional layers.

[0126] Full Resolution Pipeline (FRP):

[0127] Downsampling and upsampling are commonly used in transformer-based architectures. These operations can reduce network parameters and speed up the training process. At the same time, downsampling can provide a larger acceptance range for global feature extraction. However, information may be lost during downsampling, while upsampling will inevitably increase artifacts. Therefore, the transformer-based post-enhancement back-end network can use the full-resolution pipeline.

[0128] Given a reconstructed frame First, a convolutional layer is applied to obtain the low-level feature embedding F0∈ Where H×W represents the spatial height and width, and C represents the number of channels. Then, the feature F0 undergoes six transformation blocks to obtain the refined feature F R Finally, for F R Apply the convolutional layer to get the residual image Reconstruct the frame Added to the residual image, we get Ablation studies show that adding upsampling and downsampling operations in the pipeline significantly degrades compression performance, which validates the advantages of the proposed full-resolution pipeline.

[0129] Transpose Gated Transformer Block (TGTB):

[0130] As the basic unit of the post-enhancement network, the Transposed Gated Transformer Block (TGTB) is designed without layer normalization, which includes Multi-Head Transposed Attention (MTA) and Gated Feedforward Network (GFN). The details of TGTB are as follows.

[0131] Is there a normalization layer?

[0132] First, the normalization layer of the transformer block is removed to prevent performance degradation in learning-based video compression tasks. Layer normalization is usually used to normalize the intermediate layer distribution in the transformer to speed up model convergence by losing unimportant information

[63] . However, the literature

[64] points out that different tasks prefer different normalization methods, and inappropriate normalization methods may lead to performance degradation. Since the transformation-based enhancement network is incorporated into the entire compression framework and all modules are trained end-to-end, the unimportant information lost by layer normalization may be recovered by other modules. In this case, the entire framework will be deceived, and unimportant information will be lost by normalization, and at the same time, the information will be recovered by other modules, resulting in performance degradation. Therefore, in this embodiment, the transformer module is designed without layer normalization. The impact of layer normalization is demonstrated in ablation studies, verifying the benefits of removing the normalization layer.

[0133] ·MTA and GFN

[0134] Second, as mentioned before, the main computational cost of the Transformer comes from the Transformer’s self-attention layer. Fig. 9 As shown, in one embodiment, multi-head transposed attention (MTA) is applied to alleviate the computational problem. In this embodiment, the key is to calculate self-attention across channels rather than spatial dimensions. By calculating the cross covariance of each channel, a transposed attention map for global context encoding is generated. In addition, before calculating the feature covariance, deep convolution is applied to emphasize local context.

[0135] According to the feature input MTA generates query (Q), key (K), and value (V) projections with local context. Specifically, 1×1 convolution aggregates pixel cross-channel context, while 3×3 depthwise convolution encodes channel-space context. Next, the query (Q) is reshaped into The key (K) is reshaped to Therefore, the dot product interaction between the query and the key is a Instead of a transposed attention map of size In general, the whole process of MTA can be defined as follows:

[0136]

[0137]

[0138]

[0139]

[0140]

[0141] Among them, X and are the input and output feature maps, is a 1×1 pointwise convolution, Represents a 3×3 depthwise convolution. and is an unbiased convolution. In from the original is obtained after reshaping. α is a learnable parameter that controls the Similar to multi-head self-attention [6], the number of channels is divided into multiple heads and different attention maps are learned in parallel.

[0142] In addition, if Fig.10 As shown in Figure 1, the gated feedforward network also uses a gating mechanism and depthwise convolution. The gating mechanism is implemented by the element-wise product of two parallel path transformation layers, one of which is activated by the GELU nonlinearity

[65] . The depthwise convolution is used to obtain information from spatially adjacent pixel locations. The overall process of GFN is as follows:

[0143]

[0144]

[0145] where X and ⊙ represents element-wise multiplication. φ represents the gating function, is a 1×1 pointwise convolution, and represents 3×3 depthwise convolutions, all of which are unbiased convolutions.

[0146] D. Loss Function

[0147] The goal of the loss function is to optimize the distortion rate (RD) cost of the entire framework. The loss function is defined as follows

[0148]

[0149] in, is the input frame and the reconstructed frame The mean squared error (MSE) or multi-scale structural similarity (MSSSIM)

[66] is used to evaluate the distortion. is the number of bits consumed to encode the quantized motion vector potential and the corresponding prior value. f is the number of bits consumed to encode the quantized context potential and the corresponding prior value. (*) is given by the cross entropy between the marginal distribution of the latent code and the estimated distribution

[67] . In addition, λ controls the trade-off between rate R and distortion D.

[0150] Experimental results and analysis

[0151] A. Implementation Details

[0152] Training dataset: According to one embodiment, Vimeo-90k

[68] is used as training data, which is commonly used for learning-based video compression tasks. The dataset consists of 91,701 sequences with a fixed resolution of 448×256 and each sequence contains 7 frames. The video sequences are randomly cropped into 256×256 segments.

[0153] Test dataset: The test dataset includes HEVC standard sequences from common test conditions used by the standards community

[69] . Specifically, Class B (1920×1080 resolution), Class C (832×480 resolution), Class D (416×240 resolution), and Class E (1280×720 resolution) are used to evaluate the performance. Since previous learning-based codecs do not evaluate HEVC Class A sequences (2560×1600 resolution), the test is not performed on HEVC Class A sequences (2560×1600 resolution)

[13] ,

[15] ,

[42] . In addition, 1920×1080 resolution videos from the UVG

[70] and MCL-JCV

[71] datasets are also tested. In total, the compression performance is measured for 16 HEVC sequences, 7 UVG sequences, and 30 MCL-JCV sequences, including slow / fast motion, homogeneous / non-homogeneous scenes, object rotation, complex textures, etc.

[0154] Implementation details: The example model is implemented on an NVIDIA 3090 GPU using PyTorch

[72] and the CompressAI

[73] project. For comparison purposes, this example uses the DCVC

[15] method and trains four models with different λ values ​​(λ equals 256, 512, 1024, 2048 for PSNR and 8, 16, 32, 64 for MS-SSIM). The AdamW

[74] optimizer is used, the initial learning rate is 1e-4, and the batch size is set to 8.

[0155] Test configuration settings: According to the settings in

[15] , the group of pictures (GOP) of the HEVC sequence is set to 10, and the group of pictures (GOP) of other sequences is set to 12. In addition, this example tests 100 frames for the HEVC sequence and 120 frames for other sequences. Since the focus of the present invention is inter-frame coding, the existing learning-based image compression model in the CompressAI project

[73] is used for intra-frame coding. In this example, cheng2020-anchor

[12] is used for the PSNR target and hyperprior

[27] is used for the MS-SSIM target. Under different bit rate encoding conditions, the quality levels of cheng2020-anchor and hyperprior are set to 3, 4, 5 and 6, respectively, where the larger the level value, the better the frame quality, and in learning-based inter-frame coding, the corresponding λ is larger.

[0156] B. Performance Evaluation

[0157] The proposed DEVC method embodiment is compared with existing codecs such as H.264[1], H.265

[54] , and VVC[3] to demonstrate its coding efficiency. One of the embodiments applies x265 very slow profile and x264 very fast profile for H.265 and H.264 in FFmpeg, as follows.

[0158] ·ffmpeg-y pix_fmt yuv420p-s WX H-rFR-i input.yuv-vframes Nc:vlibx265-tnne zerolatency-preset verys-low-x265-params "crf=QP:keint=OOP:verbose=1" out-put.mkv

[0159] ·ffmpeg-y pix_fmt yuv420p-s WX Hr FR-i input.yuv-vframes Nc: vlibx264-tune zerolatency-preset very-fast-crf QP-g GOP-sc_threshold0output.mkv

[0160] Among them, W, H, FR, N, QP and GOP represent height, width, frame rate, number of encoded frames, quantization parameter and a group of pictures respectively. The internal color space is YUV420, and QP is set to 15, 19, 23 and 27 for encoding at different bit rates. It is worth mentioning that the test configuration settings for H.264 and H.265 have not changed. The GOP of the HEVC sequence is set to 10, and the GOP of other sequences is set to 12. HEVC is tested for 100 frames, and other sequences are tested for 120 frames. In addition, W, H and FR are configured according to the properties of the test sequence.

[0161] Regarding VVC, VTM-14.0 is considered as a competitive benchmark for compression test data sets. Since the GOP size of the encoder low-delay VTM configuration is 8, it only supports cycles that are multiples of 8. At the same time, the method of this embodiment is a predictive coding framework. Therefore, the cycle is set to 16, and the predictive configuration lowdely_p_vtm.cfg is selected as the default configuration file of VVC for performance comparison.

[0162] ·EncoderAppStatic-c encoder_lowdelay_P_vtm.cfg--

[0163] InputFile=input.yuv--BitsstreamFile=rec.bin--Decoding

[0164] RefreshType=2--InputBitDepth=8--OutputtBitDepth=8

[0165] --OutputBitDepthC=8--FrameRate=FR--FramesToBe

[0166] Encoded=N--SourceWidth=W--SourceHeight=H--

[0167] IntraPeriod=16--QP=QP--Level=6.2--ReconFile=

[0168] rec.yuv

[0169] Where W, H, FR, N and QP are height, width, frame rate, number of coded frames and quantization parameter respectively. The QP of VVC is set to 22, 25, 28 and 31. In addition, HEVC is tested with 100 frames and other sequences are tested with 120 frames.

[0170] Since neural networks are trained and executed in the RGB domain for vision tasks, some existing learning-based video compression works use the RGB format. This example also evaluates the frame quality (PSNR or MS-SSIM) in the RGB domain. For H.264, H.265, and VVC, the yuv file is converted to png format images and the corresponding frame quality is calculated.

[0171] At the same time, representative learning-based video coding methods are selected as baselines, including DVC

[13] , FVC

[42] , and DCVC

[15] . DVC is the first end-to-end predictive coding framework and pioneered learning-based video compression. FVC shifts the pixel space framework to the feature space video compression framework. DCVC implements a conditional coding framework that uses high-dimensional context as a condition to guide compression instead of calculating the residual between the predicted frame and the input frame. In general, DVC, FVC, and DCVC have made great improvements in learning-based video compression and are considered to be representative learning-based methods with SOTA performance.

[0172] In addition, Li et al. proposed DCVC in

[15] and further improved DCVC

[49] by hybrid spatiotemporal entropy modeling and content-adaptive quantization mechanism, and its performance is comparable to VVC. In

[49] , the features of the decoded frame and the reconstructed frame before the temporal context are propagated for inter-frame prediction. In contrast, in one embodiment of the present invention, only the decoded frame is propagated. The method proposed in this paper can be used as another method to improve compression efficiency and can be extended to other learning-based methods on top of conditional coding. In addition, Lin et al. proposed multi-frame prediction in

[50] to generate and propagate multiple MV fields. The learning-based multi-reference framework

[52] ,

[53] has also been proposed for video compression. In this example, considering that only one decoded frame is used as a reference for fair comparison, only the performance comparison between the proposed DEVC implementation and VVC, DVC, FVC, DCVC, H.265, and H.264 is shown.

[0173] PSNR indicator evaluation:

[0174] BD-Rate

[75] is used to measure the performance of the proposed method (DEVC) and other state-of-the-art methods. Taking the very slow profile of x265 as the anchor, the PSNR comparison of BD-Rate is shown in Table 1. Negative numbers represent bitrate savings. The best and second-best neural video coding methods are bolded and underlined in brackets. In addition, the encoding performance curves are visualized with bits per pixel (bpp) as the horizontal axis and the reconstructed PSNR as the vertical axis. Figure 11a-Figure 11f The bpp and PSNR of each category are averaged.

[0175] Table 1 Comparison of BD-Rates from the perspective of PSNR (frame interval 101 / 12)

[0176]

[0177]

[0178] The anchor point is that x265 is very slow, negative numbers indicate bitrate savings.

[0179] - The best and second best neural video coding methods are marked as bold and underlined in brackets.

[0180] Compared with the previous traditional video codecs H.264 and H.265, the DEVC of the present invention achieves a significant performance gap, which shows the great potential of learning-based methods. On all test datasets, DEVC has an average bit saving rate of 36.40%, which is better than H.265, especially the bit saving rate of UVG sequence is as high as 46.33%, and the bit saving rate of HEVCB class sequence is as high as 41.06%. Compared with the VTM LDP configuration, the proposed DEVC implementation is inferior in terms of BD rate of PSNR, which can be further explained as follows. The DEVC implementation of the present invention only applies one reference frame and fixed QP for mutual prediction. However, the LDP configuration in VTM uses multiple reference frames and dynamic QP, which is a better configuration. At the same time, there are fewer I frames in the VTM test because the frame interval is set to 16 instead of 10 / 12, and the HEVC sequence is tested with 100 frames, while the other sequences are tested with 120 frames.

[0181] From the experimental results, it can be seen that the proposed DEVC solution outperforms the listed end-to-end video codecs on all test datasets, which shows that it has strong generalization ability due to the different features contained in the test datasets. In Table 2, taking DCVC as the anchor, the performance improvement after adopting the dual enhancement modeling scheme in one implementation is shown. In terms of PSNR, the proposed method reduces the bit rate by 9.85% on average.

[0182] Table 2

[0183] Comparison of BD-Rate between DEVC and DCVC from the perspective of PSNR and MS-SSIM (frame interval 10 / 12)

[0184]

[0185]

[0186] The anchor point is DCVC.

[0187] Negative numbers indicate bitrate savings.

[0188] MS-SSIM indicator evaluation:

[0189] The distortion metric is now moved from PSNR to MS-SSIM, and BD-Rate comparison is performed. Figure 12a-12f As shown in Figure 2, the bpp and MS-SSIM averages of each test dataset category were calculated and the RD curves were plotted. Figure 12a-12f In the table, bpp and MS-SSIM are the average values ​​for each category. The detailed performance results are shown in Table 3. Compared with H.265, the DEVC implementation improves the RD performance by an average of 52.85% in MS-SSIM. In addition, it also saves about 6.7% bits over the latest traditional codec VVC. At the same time, the DEVC implementation outperforms other listed learning-based codecs. In Table 2, taking DCVC as the anchor point, it can be seen that the RD performance of the proposed method in this paper is improved by 12.19%.

[0190] Table 3

[0191] Comparison of BD-Rate between DEVC and DCVC from the perspective of PSNR and MS-SSIM (frame interval 10 / 12)

[0192]

[0193] The anchor point is x265 (very slow), negative numbers indicate bitrate savings.

[0194] The best and second best neural video coding methods are marked in bold and underlined in brackets.

[0195] Subjective quality comparison:

[0196] In terms of subjective quality comparison, Fig.13 The frames in RaceHorses, Cactus, and BQSquare are selected, and the local areas are enlarged for better display. Fig.13 As shown, compared with DCVC, the DEVC implementation of the present invention retains more details and has higher texture fidelity.

[0197] C. Complexity Analysis

[0198] The model complexity of the number of parameters, MAC (multiplication-accumulation operation), and encoding-decoding time cost are compared. The complexity analysis involves 480p resolution sequences of HEVC Class C. In addition, the complexity analysis is performed on computers equipped with NVIDIA 3090 GPU and Intel (R) Xeon (R) Silver 4210 CPU @ 2.20GHz. The complexity comparison results are shown in Table 4. For the time cost of encoding and decoding a frame, the codec time is measured, including the time to write the bitstream and read the bitstream. Since the time to encode and decode a frame of image is affected by the image content and the computer status, the 95% confidence time cost interval of DCVC and DEVC is calculated.

[0199] Table 4

[0200] Comparison of complexity between the present invention (DEVC) and DCVC

[0201]

[0202] The test used HEVC Level C 480p resolution sequence.

[0203] Furthermore, δ is defined as follows:

[0204]

[0205] Among them, T DEVC and T DCVC The total encoding time of the DEVC implementation scheme of this embodiment and DCVC is shown. Compared with DCVC

[15] , the time cost of this embodiment is slightly increased by only 2.12%. The number of parameters increases from 7.94M to 8.5M, an increase of 7.05%, while the MAC increases from 429.93G to 635.72G.

[0206] The self-attention structure can capture long-distance dependencies well, but it requires a lot of computation. According to one embodiment, a transposed gated transform block (TGTB) is proposed to reduce the main computational overhead of self-attention, and it is worth exploring how to further reduce the amount of computation. After adopting the proposed design, the compression efficiency is improved. Under the default GOP setting, the average values ​​of PSNR and MS-SSIM are 9.85% and 12.19%, respectively. Specifically, when the GOP size is 32, the PSNR can be improved by 19.28%, as shown in Table 5.

[0207] D. Ablation studies

[0208] Effects of different components:

[0209] In order to demonstrate the effectiveness of the enhanced context mining model (ECM) and the transformer-based post-enhancement back-end network (TPEN), the above-mentioned DEVC implementation scheme of the present invention is used as a baseline. Table 6 lists the impact of different components. A positive number indicates a decrease in compression performance. As shown in Table 6, using ECM to eliminate redundancy across context channels can bring an 11% gain, and the post-enhancement back-end network can bring an additional 3% performance gain. It can be verified that the ECM proposed in this paper can better learn the context, while the transformer-based post-enhancement network further improves the compression performance.

[0210] Table 5

[0211] BD-Rate comparison of PSNR between DEVC and DCVC, including default GOP

[0212] And the comparison when GOP=32

[0213]

[0214] • The anchor point is DCVC (default GOP), and DCVC GOP=32.

[0215] Negative numbers indicate bitrate savings.

[0216] Table 6 Comparison of effectiveness of different components in the present invention

[0217]

[0218] The baseline represents the most preferred DEVC proposed in this paper

[0219] A positive number indicates a decrease in compression performance.

[0220] Ablation studies with full resolution pipeline:

[0221] In order to study the benefits of the full-resolution pipeline for TPEN, up- and down-sampling are added to the pipeline, and the performance comparison is shown in Table 7. As can be seen from Table 7, after adding down / up sampling to the pipeline, the compression performance decreases by 2.29%. This experimental result is consistent with the original intention of the design, that is, information may be lost during downsampling, and upsampling will inevitably increase artifacts, and also verifies the advantages of the proposed full-resolution pipeline.

[0222] Table 7 Ablation study of the full resolution pipeline

[0223]

[0224] The baseline represents the most preferred DEVC proposed in this paper

[0225] A positive number indicates a decrease in compression performance.

[0226] Benefits of removing the normalization layer in TPEN:

[0227] Taking the BQTerrace sequence in HEVC ClassB (1920x1080) as an example, the bpp / PSNR values ​​with and without normalization layers in TPEN are introduced. The impact of the normalization layer is shown in Table 8. It can be seen that after layer normalization in TPEN, the bpp value is larger or the quality is poor. This phenomenon can be explained as follows. Since the transform-based enhancement network is incorporated into the entire compression framework and all modules are trained end-to-end, the entire framework will be fooled and lose unimportant information through normalization while recovering information through other modules at the same time, resulting in performance degradation. Therefore, a transformer module without layer normalization is designed.

[0228] Table 8 Effect of normalization layer in TPEN

[0229]

[0230] BQTerrace sequence is used as an example

[0231] Ablation study of the transposed gated converter module:

[0232] Table 9 shows the ablation study of the Transpose Gated Transformer block. Without MTA, the performance drops by 1.43%, and without GFN, the performance drops by 2.82%.

[0233] Table 9

[0234] Ablation study of the transposed internally controlled converter block

[0235]

[0236] A positive number indicates a decrease in compression performance.

[0237] E. Error Propagation Analysis

[0238] Take HEVC Class C RaceHorses as an example. Figure 14a to Figure 14h The PSNR and bit rate comparison of DCVC and the proposed DEVC implementation are shown. It can be seen that under all λ settings, the DEVC implementation can achieve better reconstruction quality at a lower bit cost. In summary, the DEVC implementation can effectively reduce error propagation, and the advantages of the DEVC implementation are more obvious when the GOP size is larger.

[0239] in conclusion

[0240] According to some embodiments of the present invention, a dual enhancement modeling for learning video compression is designed. In particular, a context mining model is designed to reduce redundancy across context channels. Residual learning and convolution operations are applied with context channels to reduce artifacts caused by bilinear distortions and inaccurate motion estimation. In addition, a transformer-based post-enhancement back-end network is designed to capture long-range dependencies. Specifically, it provides a full-resolution pipeline for the post-enhancement network without information loss. In addition, a normalization-free transposed gated transformer block is designed to alleviate self-attention calculations and make the entire framework applicable to high-resolution frames.

[0241] Extensive experiments show that the proposed scheme saves about 6.7% of bits compared to VVC and 12.19% of MS-SSIM compared to DCVC. In terms of PSNR indicators, the proposed scheme exceeds HEVC, with an average improvement of 36.40%, UVG ​​sequences saving 46.33% of bits, and HEVCB-like sequences saving 41.06% of bits. Compared with DCVC, the proposed scheme saves 9.85% of bits. Specifically, when the GOP is 32, the PSNR is improved by 19.28%. When the GOP size increases, the advantages of the proposed method / framework will be more obvious, and the time complexity increment of the proposed model will be negligible.

[0242] system

[0243] Fig.15 An example information processing system 1500 is shown, which can be used to implement embodiments of the present invention (including Figure 2 , 3, 4, 5, 6, 8, 9 and 10) for learning one or more methods of video compression. The information processing system 1500 generally includes appropriate components required to receive, store and execute appropriate computer instructions, commands and / or codes. The main components of the information processing system 1500 are the processor 1502 and the memory (storage) 1504. The processor 1502 may include one or more: CPU, MCU, GPU, logic circuit, RaspberryPi chip, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA) or any other digital or analog circuit / circuit that is configured to interpret and / or execute program instructions and / or process signals and / or information and / or data. The memory 1504 may include one or more volatile memories (such as RAM, DRAM, SRAM, etc.), one or more non-volatile memories (such as ROM, PROM, EPROM, EEPROM, FRAM, MRAM, FLASH, SSD, NAND, NVDIMM, etc.) or any combination thereof. Appropriate computer instructions, commands, codes, information and / or data may be stored in the memory 1504. Computer instructions for executing or facilitating the execution of the method embodiments of the present invention may be stored in the memory 1504. The processor 1502 and the memory (storage) 1504 may be integrated or separated (and operably connected). Optionally, the information processing system 1500 also includes one or more input devices 1506. Examples of such input devices 1506 include: keyboard, mouse, stylus, image scanner, microphone, tactile / touch input device (such as touch sensitive screen), image / video input device (such as camera), etc. Optionally, the information processing system 1500 also includes one or more output devices 1508. Examples of such output devices 1508 include: display (such as monitor, screen, projector, etc.), speaker, earphone, headset, printer, additive manufacturing machine (such as 3D printer), etc. The display may include an LCD display, an LED / OLED display or other suitable display, which may be touch sensitive or non-touch sensitive. The information processing system 1500 may further include one or more disk drives 1512, which may include one or more of a solid-state drive, a hard disk drive, an optical drive, a flash drive, a tape drive, etc. A suitable operating system may be installed in the information processing system 1500, for example, on the disk drive 1512 or in the memory 1504. The memory 1504 and the disk drive 1512 may be operated by the processor 1502. Optionally, the information processing system 1500 further includes a communication device 1510 for establishing one or more communication links (not shown) with one or more other computing devices (such as a server, a personal computer, a terminal, a tablet computer, a mobile phone, a watch, an Internet of Things device, or other wireless computing devices).The communication device 1510 may include one or more of the following devices: a modem, a network interface card (NIC), an integrated network interface, an NFC transceiver, a ZigBee transceiver, a Wi-Fi transceiver, etc. Transceiver, radio frequency transceiver, cellular (2G, 3G, 4G, 5G, 5G or above or similar) transceiver, optical port, infrared port, USB connection or other wired or wireless communication interface. The transceiver can be implemented by one or more devices (integrated transmitter and receiver, independent transmitter and receiver, etc.). The communication link can be wired or wireless and is used to communicate commands, instructions, information and / or data. In one example, the processor 1502, memory 1504 (optionally input device 1506, output device 1508, communication device 1510 and disk drive 1512 (if any)) are directly or indirectly connected to each other via a bus, peripheral component interconnect (PCI) (such as PCI Express), universal serial bus (USB), optical bus or other similar bus structure. In one embodiment, at least some of these components can be connected wirelessly, for example through a network, such as the Internet or a cloud computing network. It will be appreciated by those skilled in the art that Fig.15 The information processing system 1500 shown is only an example, and in other embodiments, the information processing system 1500 may have a different configuration (eg, include more components, have fewer components, etc.).

[0244] Although not required, one or more embodiments described with reference to the accompanying drawings may be implemented as an application programming interface (API) or a series of libraries for use by developers, or may be included in another software application, such as a terminal or computer operating system or a portable computing device operating system. In one or more embodiments, since program modules include routines, programs, objects, components, and data files that assist in performing specific functions, technicians can understand that the functionality of the software application can be distributed among multiple routines, objects, and / or components to achieve the same functionality required herein.

[0245] In addition, it is also understood that if the methods and systems of the present invention are implemented entirely or partially by a computing system, any suitable computing system architecture may be used. This includes stand-alone computers, network computers, dedicated or non-dedicated hardware devices. When the terms "computing system" and "computing device" are used, these terms are intended to include (but not limited to) any suitable arrangement of computers or information processing hardware capable of implementing the described functions.

[0246] It will be appreciated by those skilled in the art that the description and / or illustrated embodiments of the present invention may be varied and / or modified to provide other embodiments of the present invention. Therefore, the embodiments described and / or illustrated by the present invention should be considered illustrative in all respects and not restrictive. Examples of optional features of certain embodiments of the present invention are provided in the summary and description. Some embodiments of the present invention may include one or more of these optional features (some of which are not specifically described in the drawings). Some embodiments of the present invention may lack one or more of these optional features (some of which are not specifically described in the drawings).

[0247] References

[0248] [1] T. Wiegand, GJ Sullivan, G. Bjontegaard, and A. Luthra, "Overview of the H.264 / AVC video coding standard," IEEE Transactions on Circuits and Systems for Video Technology, vol.13, no.7, pp.560–576, 2003.

[0249] [2]GJSullivan, J.-R.Ohm, W.-J.Han, and T.Wiegand, "Overview of the high efficiency video coding (HEVC) standard," IEEE Transactions onCircuits andSystems for Video Technology, vol.22, no.12, pp.1649–1668, 2012.

[0251] [3]B.Bross,Y.-K.Wang,Y.Ye,S.Liu,J.Chen,GJSullivan,and JROhm,

[0252] "Overview of the versatile video coding (VVC) standard and its applications," IEEE Transactions on Circuits and Systems for Video Technology, vol.31, no.10, pp.3736–3764, 2021.

[0253] [4]S.Ma,X.Zhang,C.Jia,Z.Zhao,S.Wang,and S.Wang,“Image andvideocompression with neural networks:A review,”IEEE Transactions onCircuitsand Systems for Video Technology,vol.30,no.6,pp.1683–1698,2019.

[0255] [5]K.He,X.Zhang,S.Ren,and J.Sun,“Deep residual learning for imagerecognition,”in Proceedings of the IEEE Conference on Computer Vision andPattern Recognition,2016,pp.770–778.

[0256] [6]A.Dosovitskiy,L.Beyer,A.Kolesnikov,D.Weissenborn,X.Zhai,T.Unterthiner,M.Dehghani,M.Minderer,G.Heigold,S.Gelly et al.,“An image isworth 16x16 words:Transformers for image recognition at scale,”inInternational Conference on Learning Representations,2020.

[0257] [7]Z.Liu,Y.Lin,Y.Cao,H.Hu,Y.Wei,Z.Zhang,S.Lin,and B.Guo,“SwinTransformer:Hierarchical vision transformer using shifted windows,”inProceedings of the IEEE International Conference on Computer Vision,2021,pp.10012–10022.

[0258] [8]D.Liu,Y.Li,J.Lin,H.Li,and F.Wu,“Deep learning-based video coding:Areview and a case study,”ACM Computing Surveys,vol.53,no.1,pp.1–35,2020.

[0260] [9]D.Ding,Z.Ma,D.Chen,Q.Chen,Z.Liu,and F.Zhu,“Advances in videocompression system using deep neural network:A review and case studies,”Proceedings of the IEEE,vol.109,no.9,pp.1494–1520,2021.

[0261]

[10] Y.Zhang,S.Kwong,and S.Wang,“Machine learning based video codingoptimizations:A survey,”Information Sciences,vol.506,pp.395–423,2020.

[0262]

[11] J.Balle,V.Laparra,and E.P.Simoncelli,“End-to-end optimized image′compression,”in International Conference on Learning Representations,2017.

[0263]

[12] Z.Cheng,H.Sun,M.Takeuchi,and J.Katto,“Learned image compressionwith discretized gaussian mixture likelihoods and attention modules,”inProceedings of the IEEE Conference on Computer Vision and PatternRecognition,2020,pp.7939–7948.

[0264]

[13] G.Lu,W.Ouyang,D.Xu,X.Zhang,C.Cai,and Z.Gao,“DVC:An end-to-enddeep video compression framework,”in Proceedings of the IEEE InternationalConference on Computer Vision,2019,pp.11006–11015.

[0265]

[14] T.Ladune,P.Philippe,W.Hamidouche,L.Zhang,and O.Deforges,′“Opticalflow and mode selection for learning-based video coding,”in IEEEInternational Workshop on Multimedia Signal Processing,2020,pp.1–6.

[0266]

[15] J.Li,B.Li,and Y.Lu,“Deep contextual video compression,”Proceedings of Advances in Neural Information Processing Systems,vol.34,pp.18114–18125,2021.

[0267]

[16] C.D.Pham,C.Fu,and J.Zhou,“Deep learning based spatial-temporalin-loop filtering for versatile video coding,”in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition,2021,pp.1861–1865.

[0268]

[17] F.Zhang,C.Feng,and D.R.Bull,“Enhancing VVC through CNNbased post-processing,”in Proceedings of International Conference on Multimedia andExpo,2020,pp.1–6.

[0269]

[18] D.Ma,F.Zhang,and D.R.Bull,“MFRNet:A new CNN architecture forpost-processing and in-loop filtering,”IEEE Journal of Selected Topics inSignal Processing,vol.15,no.2,pp.378–387,2020.

[0270]

[19] W.Fedus,B.Zoph,and N.Shazeer,“Switch Transformers:Scaling totrillion parameter models with simple and efficient sparsity,”The Journal ofMachine Learning Research,vol.23,no.1,pp.5232–5270,2022.

[0271]

[20] Y.Liu,M.Ott,N.Goyal,J.Du,M.Joshi,D.Chen,O.Levy,M.Lewis,L.Zettlemoyer,and V.Stoyanov,“Roberta:A robustly optimized bert pretrainingapproach,”arXiv preprint arXiv:1907.11692,2019.

[0272]

[21] A.Vaswani,N.Shazeer,N.Parmar,J.Uszkoreit,L.Jones,A.N.Gomez, Kaiser,and I.Polosukhin,“Attention is all you need,”Proceedings of Advancesin Neural Information Processing Systems,vol.30,2017.

[0273]

[22] Y.Mei,Y.Fan,Y.Zhang,J.Yu,Y.Zhou,D.Liu,Y.Fu,T.S.Huang,and H.Shi,“Pyramid attention networks for image restoration,”arXiv preprint arXiv:2004.13824,2020.

[0274]

[23] S.W.Zamir,A.Arora,S.Khan,M.Hayat,F.S.Khan,and M.-H.Yang,“Restormer:Efficient transformer for high-resolution image restoration,”inProceedings of the IEEE Conference on Computer Vision and PatternRecognition,2022,pp.5728–5739.

[0275]

[24] G.K.Wallace,“The JPEG still picture compression standard,”IEEETransactions on Consumer Electronics,vol.38,no.1,1992.

[0276]

[25] C.Christopoulos,A.Skodras,and T.Ebrahimi,“The jpeg2000 stillimagecoding system:an overview,”IEEE Transactions on Consumer Electronics,

[0277] vol.46,no.4,pp.1103–1127,2000.

[0278]

[26] F.Bellard,“Bpg image fromat,”2015,[online available]athttps: / / bellard.org / bpg / .

[0279] [27 J.Balle,D.Minnen,S.Singh,S.J.Hwang,and N.Johnston,“Vari-′ationalimage compression with a scale hyperprior,”arXiv preprintarXiv:1802.01436,2018.

[0280]

[28] D.Minnen,J.Balle,and G.D.Toderici,“Joint autoregressive and′

[0281] hierarchical priors for learned image compression,”ProceedingsofAdvances in Neural Information Processing Systems,vol.31,2018.

[0282]

[29] J.Lee,S.Cho,and S.-K.Beack,“Context-adaptive entropy model forend-to-end optimized image compression,”arXiv preprint arXiv:1809.10452,2018.

[0285]

[30] F.Mentzer,E.Agustsson,M.Tschannen,R.Timofte,and L.Van Gool,

[0286] “Conditional probability models for deep image compression,”inProceedings of the IEEE Conference on Computer Vision andPatternRecognition,2018,pp.4394–4402.

[0287]

[31] Y.Hu,W.Yang,and J.Liu,“Coarse-to-fine hyper-prior modelingforlearned image compression,”in Proceedings of the AAAI ConferenceonArtificial Intelligence,vol.34,no.07,2020,pp.11013–11020.

[0288]

[32] H.Liu,Y.Zhang,H.Zhang,C.Fan,S.Kwong,C.-C.J.Kuo,and X.Fan,

[0289] “Deep learning-based picture-wise just noticeable distortionpredictionmodel for image compression,”IEEE Transactions on Image Processing,

[0290] vol.29,pp.641–656,2019.

[0291]

[33] G.Toderici,S.M.O’Malley,S.J.Hwang,D.Vincent,D.Minnen,S.Baluja,M.Covell,and R.Sukthankar,“Variable rate image compression withrecurrentneural networks,”arXiv preprint arXiv:1511.06085,2015.

[0292]

[34] G.Toderici,D.Vincent,N.Johnston,S.Jin Hwang,D.Minnen,J.Shor,andM.Covell,“Full resolution image compression with recurrent neuralnetworks,”inProceedings of the IEEE Conference on Computer Vision andPattern Recognition,2017,pp.5306–5314.

[0293]

[35] N.Johnston,D.Vincent,D.Minnen,M.Covell,S.Singh,T.Chinen,S.J.Hwang,J.Shor,and G.Toderici,“Improved lossy image compression withpriming and spatially adaptive bit rates for recurrent networks,”inProceedings of the IEEE Conference on Computer Vision and PatternRecognition,2018,pp.4385–4393.

[0294]

[36] E.Agustsson,M.Tschannen,F.Mentzer,R.Timofte,and L.V.Gool,“Generative adversarial networks for extreme learned image compression,”inProceedings of the IEEE International Conference on Computer Vision,2019,pp.221–231.

[0295]

[37] F.Mentzer,G.D.Toderici,M.Tschannen,and E.Agustsson,“Highfidelitygenerative image compression,”Proceedings of Advances in Neural InformationProcessing Systems,vol.33,pp.11913–11924,2020.

[0296]

[38] T.Zhao,Y.Huang,W.Feng,Y.Xu,and S.Kwong,“Efficient VVC intraprediction based on deep feature fusion and probability estimation,”IEEETransactions on Multimedia,2022.

[0297]

[39] L.Zhu,Y.Zhang,N.Li,G.Jiang,and S.Kwong,“Deep learning-based intramode derivation for versatile video coding,”ACM Transactions on MultimediaComputing,Communications and Applications,vol.19,no.2s,pp.1–20,2023.

[0298]

[40] G.Lu,X.Zhang,W.Ouyang,L.Chen,Z.Gao,and D.Xu,“An end-toendlearning framework for video compression,”IEEE Transactions on PatternAnalysis and Machine Intelligence,vol.43,no.10,pp.3292–3308,2020.

[0299]

[41] Z.Hu,Z.Chen,D.Xu,G.Lu,W.Ouyang,and S.Gu,“Improving deep videocompression by resolution-adaptive flow coding,”in Proceedings of theEuropean Conference on Computer Vision,2020,pp.193–209.

[0300]

[42] Z.Hu,G.Lu,and D.Xu,“FVC:A new framework towards deep videocompression in feature space,”in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition,2021,pp.1502–1511.

[0301]

[43] Z.Hu,G.Lu,J.Guo,S.Liu,W.Jiang,and D.Xu,“Coarse-ToFine Deep VideoCoding With Hyperprior-Guided Mode Prediction,”in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition,2022,pp.5921–5930.

[0302]

[44] E.Agustsson,D.Minnen,N.Johnston,J.Balle,S.J.Hwang,and G.Toderici,“Scale-space flow for end-to-end optimized video compression,”in Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition,2020,pp.8503–8512.

[0303]

[45] K.Lin,C.Jia,X.Zhang,S.Wang,S.Ma,and W.Gao,“DMVC:Decomposed motionmodeling for learned video compression,”IEEE Transactions on Circuits andSystems for Video Technology,2022.

[0304]

[46] H.Guo,S.Kwong,C.Jia,and S.Wang,“Enhanced motion compensation fordeep video compression,”IEEE Signal Processing Letters,2023.

[0305]

[47] F.Mentzer,G.Toderici,D.Minnen,S.-J.Hwang,S.Caelles,M.Lucic,andE.Agustsson,“VCT:A video compression transformer,”arXiv preprint arXiv:2206.07307,2022.

[0306]

[48] X.Sheng,J.Li,B.Li,L.Li,D.Liu,and Y.Lu,“Temporal context miningfor learned video compression,”IEEE Transactions on Multimedia,2022.

[0307]

[49] J.Li,B.Li,and Y.Lu,“Hybrid spatial-temporal entropy modelling forneural video compression,”in Proceedings of the ACM International Conferenceon Multimedia,2022,pp.1503–1511.

[0308]

[50] J.Lin,D.Liu,H.Li,and F.Wu,“M-LVC:Multiple frames prediction forlearned video compression,”in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition,2020,pp.3546–3554.

[0309]

[51] R.Yang,F.Mentzer,L.Van Gool,and R.Timofte,“Learning for videocompression with recurrent auto-encoder and recurrent probability model,”IEEEJournal of Selected Topics in Signal Processing,vol.15,no.2,pp.388–401,2020.

[0310]

[52] R.Yang,F.Mentzer,L.V.Gool,and R.Timofte,“Learning for videocompression with hierarchical quality and recurrent enhancement,”inProceedings of the IEEE Conference on Computer Vision and PatternRecognition,2020,pp.6628–6637.

[0311]

[53] R.Pourreza and T.Cohen,“Extending neural p-frame codecs forbframe coding,”in Proceedings of the IEEE International Conference onComputer Vision,2021,pp.6680–6689.

[0312]

[54] V.Sze,M.Budagavi,and G.J.Sullivan,“High efficiency video coding(HEVC),”in Integrated Circuit and Systems,Algorithms andArchitectures.Springer,2014,vol.39,p.40.

[0313]

[55] Y.Dai,D.Liu,and F.Wu,“A convolutional neural network approach forpost-processing in hevc intra coding,”in MultiMedia Modeling InternationalConference,2017,pp.28–39.

[0314]

[56] Y.Zhang,T.Shen,X.Ji,Y.Zhang,R.Xiong,and Q.Dai,“Residual highwayconvolutional neural networks for in-loop filtering in HEVC,”IEEETransactions on Image Processing,vol.27,no.8,pp.3827–3841,2018.

[0315]

[57] D.Ding,L.Kong,G.Chen,Z.Liu,and Y.Fang,“A switchable deep learningapproach for in-loop filtering in video coding,”IEEE Transactions on Circuitsand Systems for Video Technology,vol.30,no.7,pp.1871–1887,2019.

[0316]

[58] C.Jia,S.Wang,X.Zhang,S.Wang,J.Liu,S.Pu,and S.Ma,“Content-awareconvolutional neural network for in-loop filtering in high efficiency videocoding,”IEEE Transactions on Image Processing,vol.28,no.7,pp.3343–3356,2019.

[0317]

[59] Z.Pan,X.Yi,Y.Zhang,B.Jeon,and S.Kwong,“Efficient in-loopfiltering based on enhanced deep convolutional neural networks for HEVC,”IEEETransactions on Image Processing,vol.29,pp.5352–5366,2020.

[0318]

[60] A.Ranjan and M.J.Black,“Optical flow estimation using a spatialpyramid network,”in Proceedings of the IEEE Conference on Computer Vision andPattern Recognition,2017,pp.4161–4170.

[0319]

[61] Y.Tian,Y.Zhang,Y.Fu,and C.X.Tdan,“Temporally-deformable alignmentnetwork for video super-resolution,”in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition,2020,pp.3357–3366.

[0320]

[62] Y.Wu and K.He,“Group normalization,”in Proceedings of theEuropean Conference on Computer Vision,2018,pp.3–19.

[0321]

[63] J.Xu,X.Sun,Z.Zhang,G.Zhao,and J.Lin,“Understanding and improvinglayer normalization,”Proceedings of Advances in Neural Information ProcessingSystems,vol.32,2019.

[0322]

[64] P.Luo,R.Zhang,J.Ren,Z.Peng,and J.Li,“Switchable normalization forlearning-to-normalize deep representation,”IEEE Transactions on PatternAnalysis and Machine Intelligence,vol.43,no.2,pp.712–728,2019.

[0323]

[65] D.Hendrycks and K.Gimpel,“Gaussian error linear units,”arXivpreprint arXiv:1606.08415,2016.

[0324]

[66] Z.Wang,E.P.Simoncelli,and A.C.Bovik,“Multiscale structuralsimilarity for image quality assessment,”in Asilomar Conference on Signals,Systems and Computers,vol.2,2003,pp.1398–1402.

[0325]

[67] J.Balle,V.Laparra,and E.P.Simoncelli,“End-to-end optimization′ofnonlinear transform codes for perceptual quality,”in Picture CodingSymposium,2016,pp.1–5.

[0326]

[68] T.Xue,B.Chen,J.Wu,D.Wei,and W.T.Freeman,“Video enhancement withtask-oriented flow,”International Journal of Computer Vision,vol.127,no.8,pp.1106–1125,2019.

[0327]

[69] F.Bossen et al.,“Common test conditions and software referenceconfigurations,”JCTVC-L1100,vol.12,no.7,2013.

[0328]

[70] A.Mercat,M.Viitanen,and J.Vanne,“UVG dataset:50 / 120fps 4ksequences for video codec analysis and development,”in Proceedings of the ACMMultimedia Systems Conference,2020,pp.297–302.

[0329]

[71] H.Wang,W.Gan,S.Hu,J.Y.Lin,L.Jin,L.Song,P.Wang,I.Katsavounidis,A.Aaron,and C.-C.J.Kuo,“MCL-JCV:A JNDbased H.264 / AVC video quality assessmentdataset,”in Proceedings of International Conference on Image Processing,2016,pp.1509–1513.

[0330]

[72] A.Paszke,S.Gross,F.Massa,A.Lerer,J.Bradbury,G.Chanan,T.Killeen,Z.Lin,N.Gimelshein,L.Antiga et al.,“Pytorch:An imperative style,high-performance deep learning library,”Proceedings of Advances in NeuralInformation Processing Systems,vol.32,2019.

[0331]

[73] J.Begaint,F.Racap′e,S.Feltman,and A.Pushparaja,“Compressai:′apytorch library and evaluation platform for end-to-end compression research,”arXiv preprint arXiv:2011.03029,2020.

[0332]

[74] I.Loshchilov and F.Hutter,“Decoupled weight decayregularization,”arXiv preprint arXiv:1711.05101,2017.

[0333]

[75] G. “Calculation of average psnr differences betweenrdcurves(vceg-m33),”in VCEG Meeting(ITU-T SG16 Q.6),2001,pp.2–4.

Claims

1. A computer-implemented method for learning video compression, comprising: processing a current frame and a previously decoded frame of video data using a motion estimation model to estimate a motion vector for each pixel; compressing the motion vector and reconstructing the motion vector into a reconstructed motion vector; Applying an enhancement context mining model to obtain enhancement context from the reconstructed motion vector and previously decoded frame features; compressing the current frame with the help of the enhancement context to obtain a reconstructed frame; as well as The reconstructed frame is provided to a post-enhancement back-end network to obtain a high-resolution frame. 2 . The computer-implemented method of claim 1 , wherein the motion estimation model is based on a spatial pyramid network.

3. The computer-implemented method of claim 1, wherein the step of applying an enhanced context mining model comprises utilizing cross-channel interactions and residual learning operations to reduce redundancy across context channels.

4. The computer-implemented method of claim 1 , wherein the step of applying an enhanced context mining model comprises: Based on the current frame and the previously decoded frame, obtaining the motion vector and the previously decoded frame feature; Warping the motion vector and the previously decoded frame feature to obtain a warped feature; as well as The warped features are processed using residual blocks and convolutional layers to obtain context.

5. The computer-implemented method of claim 4, wherein the step of applying the enhanced context mining model further comprises: combining the convolution with a rectified linear unit on the context to obtain a residual context; as well as The residual context is added to the context to obtain the enhanced context.

6. The computer-implemented method of claim 5, wherein combining the convolution with a rectified linear unit to the context comprises passing the context through multiple layers of convolution and the rectified linear unit and then through another convolution layer.

7. The computer-implemented method of claim 4, wherein the enhanced context mining model is free of a batch normalization layer.

8. The computer-implemented method of claim 1 , wherein the step of compressing the current frame comprises: Connecting the current frame and the enhancement context together; processing the current frame and the enhanced context to obtain a latent code for an entropy model; as well as With the help of the enhancement context, the latent code is converted back to pixel space to obtain the reconstructed frame.

9. The computer-implemented method of claim 1, wherein the post-enhancement backend network is transformer based.

10. The computer-implemented method of claim 1, wherein the post-augmentation back-end network comprises a plurality of transposed gated transformer blocks and a plurality of convolutional layers.

11. The computer-implemented method of claim 1 , wherein the step of providing the reconstructed frame to the post-enhancement backend network comprises: Apply convolutional layers to the reconstructed frames to obtain low-level feature embeddings Where H×W is the spatial height and width, and C represents the channel number; Process low-level features using one or more transformation blocks to obtain refined features; And applying a convolutional layer to the refined features to obtain a residual image The reconstructed frame is added to the residual image to obtain 12. The computer-implemented method of claim 11, wherein at least one of the transformer blocks comprises a transposed gated transformer block without layer normalization.

13. The computer-implemented method of claim 11, wherein at least one of the transformer blocks is modified to include a multi-head transposed attention and a gated feed-forward network.

14. The computer-implemented method of claim 13, wherein the multi-head transposed attention comprises computing self-attention across channels.

15. The computer-implemented method of claim 13, wherein multi-head transposed attention comprises applying deep convolutions.

16. The computer-implemented method of claim 13, wherein the multi-head transposed attention is derived from feature input Generates a projection of the query, keys, and values, and reshapes the query into Reshape the key into To obtain the size The transposed attention map of .

17. The computer-implemented method of claim 13, wherein: The gated feedforward network includes a gating mechanism and a deep convolution; The gating mechanism is implemented as an element-wise product of two parallel paths of conv layers, one of which is activated with a GELU nonlinearity, and The depthwise convolution is used to obtain information from spatially adjacent pixel locations.

18. A system for learning video compression, comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by one or more processors, the one or more programs comprising instructions for performing or facilitating the performance of a computer-implemented method according to claim 1.

19. A non-transitory computer readable medium having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to perform the computer implemented method of claim 1.

Citation Information

Patent Citations

  • Content adaptive dominant motion compensated prediction for next generation video coding

    CN105850133A

  • Context-based image coding and decoding

    CN115550652A

  • method and apparatus for using IoT technology to monitor elderly caregiver

    KR102188076B1

  • AI-Based Context Evaluation Engine Apparatuses, Methods and Systems

    US20190347540A1

Cited By

  • End-to-end video compression method based on temporal context mining and reconstruction frame enhancement

    CN122179567A

  • End-to-end video compression method based on temporal context mining and frame enhancement reconstruction

    CN122179567B