Neural network-based image and video compression method using conditional coding and decoding

Abstract

An image or video processing method comprising the steps of: obtaining samples of quantized residual potential values using a bitstream and an output of a first sub-network, where the first sub-network is not autoregressive; obtaining a predicted value mean using the second sub-network and the reconstructed sample points of the quantized potential values; and reconstructing a next sample point of the quantized potential value using the quantized residual sample point and the predicted mean.

Description

[0001] Cross-references to related patent applications

[0002] This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 389,784, filed on July 15, 2022, the teachings and disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] This patent application relates to the generation, storage and consumption of digital audio-visual media information in file format. Background Art

[0004] Digital video consumes the largest amount of bandwidth used on the Internet and other digital communications networks. As the number of networked user devices capable of receiving and displaying video increases, the bandwidth demands for digital video usage are likely to continue to grow. Summary of the invention

[0005] The disclosed aspects / embodiments provide techniques related to a neural network-based image and video compression method including an autoregressive subnetwork and an entropy coding and decoding engine, wherein entropy coding and decoding are performed independently of the autoregressive subnetwork.

[0006] The first aspect relates to an image or video processing method, comprising the steps of: using a bit stream and an output of a first sub-network to obtain a quantized residual potential value where the first sub-network is not autoregressive; using the second sub-network and the quantized potential value The reconstructed sample points are used to obtain the predicted value mean; and the quantized residual sample points are used And the predicted mean to reconstruct the quantitative potential value The next sample point.

[0007] The second aspect relates to a video processing method, comprising the following steps: firstly transforming an input image using an analysis transform to obtain a potential value y; using a second sub-network and a quantized potential value The predicted value mean is obtained by using the reconstructed sample points (when available) of the potential value; the predicted value is subtracted from the sample points of the potential value to obtain the residual potential sample point w; the residual potential sample point is quantized and added to this predicted value to obtain the quantitative potential value and using the first subnetwork and quantized residual potential samples The bit stream is obtained by sampling points of, wherein the first sub-network is not autoregressive.

[0008] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: Application context model.

[0009] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the output of the context model is the input of the prediction fusion module.

[0010] Optionally, in any of the foregoing aspects, another embodiment of this aspect provides: the context submodule includes: a convolution layer with a convolution kernel size of 4×3, the number of input channels is 192, the number of output channels is 384, and the input of the context submodule is And this output is fed to the prediction fusion submodule.

[0011] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the output of the context submodule and the prediction fusion submodule is a mean μ, which includes forecast or estimate.

[0012] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the context submodule and the prediction fusion submodule are jointly used to generate the prediction μ.

[0013] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the number of sample points regarded as input is proportional to the following: the size of the context submodule convolution kernel and the number of input channels of the context submodule convolution kernel.

[0014] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the number of convolution kernel sizes is 4×3, which is 4 samples in the horizontal direction multiplied by 3 samples in the vertical direction, and wherein the number of input channels is 192.

[0015] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: increasing the number of input channels or the size of the convolution kernel will increase the number of channels used to predict subsequent samples. The convolution kernel size includes the product of the size in the horizontal direction and the size in the vertical direction.

[0016] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: increasing the number of the output channels of the context submodule will increase the information obtained from the input samples, thereby improving prediction performance.

[0017] Optionally, in any of the foregoing aspects, another embodiment of this aspect provides: the number of output channels of the convolution kernel of the context submodule is 384, and wherein increasing the number of output channels causes the prediction μ to be closer to

[0018] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the convolution kernel size is represented by K×L and the input / output channels are represented by A and B respectively, and wherein increasing any one of A, B, K or L causes the complexity of the context module to increase proportionally.

[0019] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: the context submodule is divided into two parallel convolution kernels.

[0020] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the convolution kernel size of the first convolution layer is 3×2, and the size of the second convolution layer is 4×3, and wherein for both the first convolution layer and the second convolution layer, the number of input channels and the number of output channels are 192.

[0021] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the outputs of the two convolutions are connected by a concat block to generate an output of the 384 layer.

[0022] Optionally, in any of the preceding aspects, another embodiment of this aspect provides: the concat block is configured to stack two input channels on top of each other, and wherein the number of channels at the output of the concat block is the sum of the number of channels in the input of the concat block.

[0023] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: the prediction network includes the context submodule and the prediction fusion network.

[0024] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the input potential sample points are processed by at least two convolutional layers, the convolution kernel sizes of the two convolutional layers are O×P and M×N respectively, and wherein the convolution kernel sizes are different in at least one direction so that O is not equal to M or P is not equal to N.

[0025] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the outputs of the two convolutional layers are connected and input into a prediction fusion network, and wherein the prediction fusion network outputs a predicted value μ as an estimate of a potential sample point.

[0026] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the input potential sample points are processed by a first convolution layer with a convolution kernel size of M×N, and the output of the first convolution layer is processed by at least a second convolution layer with a convolution kernel size of K×L.

[0027] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the output of the first convolutional layer and the output of the second convolutional layer are connected, wherein the output is fed to the prediction fusion module, and wherein the output of the prediction fusion submodule is μ, and μ is an estimate of a potential sample point.

[0028] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the number of sample points considered by the second convolutional layer is greater than the number of sample points considered by the first convolutional layer, and wherein the number of sample points considered by the convolutional layer is called the receptive field.

[0029] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the complexity of the context model is proportional to (K×L+M×N) rather than proportional to (K+M)×(L+N).

[0030] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the potential sample point is fed to the first convolutional layer, where the residual latent samples is fed to a second convolutional layer, wherein the outputs of the first convolutional layer and the second convolutional layer are connected using the concat block and fed to the prediction fusion module, and wherein the output of the prediction fusion submodule is the μ as an estimate of the potential sample point.

[0031] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: the potential sample point and the residual potential sample point are both related to the potential sample point to be predicted, and wherein the potential sample point and the residual potential sample point are both used as inputs of a predictor.

[0032] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the output of the prediction fusion module is a potential sample point to be predicted Instead of the μ as an estimate of the potential sample, and wherein the potential sample to be predicted is obtained by combining the predicted μ with the residual sample obtained by adding or directly serving as the output of the prediction fusion module.

[0033] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: is a matrix or tensor comprising a plurality of potential samples, and wherein the input comprises samples that have been obtained by prediction.

[0034] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the prediction module includes the context module and the prediction fusion module, and wherein the prediction module is an autoregressive neural network, and the autoregressive neural network sequentially processes the potential values ​​in order Sample points.

[0035] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the fusion method includes addition.

[0036] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: performing arithmetic encoding and decoding independently of the first sub-network, and wherein the first sub-network is an autoregressive sub-network.

[0037] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: a fully sequential arithmetic encoding and / or decoding process is performed by a central processing unit.

[0038] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the second sub-network is configured to estimate probability parameters used by arithmetic coding and / or decoding process.

[0039] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the arithmetic coding and / or decoding process is used to encode and / or decode the quantized potential residual instead of the quantized potential value.

[0040] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the autoregressive subnetwork is used only to estimate the mean of the potential value rather than for estimating the mean and variance of the Gaussian distribution, and wherein the arithmetic encoder and decoder subsequently use only the mean of the potential value to encode and decode the sample points of the quantized potential value.

[0041] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the entropy decoding and / or arithmetic decoding process is performed independently, and wherein the decoding process is performed by a central processing unit.

[0042] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides that: after entropy decoding is completed, the quantized residual potential value is transmitted to a graphics processing unit.

[0043] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the context and entropy parameter module is executed independently of arithmetic encoding and decoding, and wherein the context and entropy parameter module uses a graphics processing unit.

[0044] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: idle processing time is eliminated.

[0045] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: a central processing unit and a graphics processing unit are used in coordination.

[0046] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the decoding process is performed as follows: performing an entropy decoding process to obtain all samples of the quantized potential residual; transmitting the obtained data to a graphics processing unit, wherein the transmission is performed only once; executing a context and entropy parameter module by the graphics processing unit, wherein no idle waiting occurs in the graphics processing unit, and wherein the quantized potential value is obtained; and performing the synthetic transform at the decoder to obtain the reconstructed image.

[0047] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: utilizing more than one sub-network as a super encoder or super decoder for super information.

[0048] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: generating super information using at least one sub-network, and wherein the super information depends on the parsing process for the potential information.

[0049] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: generating super information using at least one sub-network, and wherein the super information does not depend on the parsing process for the latent information.

[0050] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: using at least one sub-network to generate super information, and wherein the super information is used to predict the potential signal.

[0051] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the super information includes statistical information or probability distribution information of the potential signal, and wherein the potential signal is quantized.

[0052] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the statistical information or probability distribution information includes the mean of the potential signal.

[0053] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the statistical information or probability distribution information includes the variance of the potential signal.

[0054] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides that: the potential signal can be encoded and decoded in a predictive manner.

[0055] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: encoding y'=y-p in the encoder, where y is a potential sample and p is the prediction.

[0056] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: reconstructing y*=y'+p in the decoder.

[0057] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: y is quantized before the prediction process.

[0058] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: y is not quantized before the prediction process.

[0059] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: quantizing p before the prediction process.

[0060] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: p is not quantized before the prediction process.

[0061] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: quantizing y' after the prediction process.

[0062] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: y' is not quantized after the prediction process.

[0063] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: at least one sub-network can be used to generate the prediction p.

[0064] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides that: at least one previously decoded y* or y' can be used to generate the prediction p for the current y or y*.

[0065] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: using the quantified potential value and synthesizing the transform to obtain the reconstructed image.

[0066] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the first sub-network and the second sub-network are used as super decoders for super information.

[0067] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: at least one of the first sub-network and the second sub-network is used to generate super information, and wherein the super information depends on a parsing process for latent information.

[0068] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: at least one of the first sub-network and the second sub-network is used to generate super information, and wherein the super information does not rely on a parsing process for underlying information.

[0069] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: at least one of the first sub-network and the second sub-network is used to generate super information, and wherein the super information is used to predict potential signals.

[0070] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the super information includes statistical information or probability distribution information of the potential signal, and wherein the potential signal is quantized.

[0071] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the statistical information or probability distribution information includes the mean of the potential signal.

[0072] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the statistical information or probability distribution information includes the variance of the potential signal.

[0073] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: encoding and decoding the potential signal in a predictive manner, and wherein the potential sample points encoded and decoded in the predictive manner are represented as y'.

[0074] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: encoding and decoding the potential signal according to y'=yp, where y is a potential sample and p is a prediction.

[0075] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: reconstructing the potential signal according to y*=y'+p, wherein y* represents the reconstructed potential signal.

[0076] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: quantizing the potential sample points before the prediction.

[0077] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: the potential sample points are not quantized before the prediction.

[0078] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides

[0079] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: performing the prediction before quantization.

[0080] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: not performing the prediction before quantization.

[0081] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: not performing the prediction before quantization.

[0082] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: the potential samples encoded and decoded in the predictive manner are quantized after the prediction.

[0083] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: the potential samples encoded and decoded in the predictive manner are not quantized after the prediction.

[0084] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: generating the prediction using at least one subnetwork including a first subnetwork and a second subnetwork.

[0085] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: using the reconstructed potential signal or at least one of the potential sample points encoded and decoded in a predictive manner to generate a prediction for the current reconstructed potential signal or to generate a prediction for the current potential sample point encoded and decoded in a predictive manner.

[0086] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the first sub-network takes the first quantized super potential value as input and generates a probability parameter.

[0087] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: obtaining the samples of the quantized residual potential value includes entropy decoding, wherein the probability parameters and the bitstream are used as input.

[0088] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: obtaining the bitstream includes entropy coding, wherein the probability parameters and quantized residual potential values ​​are used as input.

[0089] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the probability parameter does not include a mean.

[0090] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: using zero mean probability distribution in entropy encoding or entropy decoding.

[0091] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides that: in addition to the reconstructed samples of the quantized potential value, the second sub-network takes the second quantized super potential value as input.

[0092] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the first quantified super potential value and the second quantified super potential value are the same.

[0093] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: obtaining the quantized super potential value from a bit stream in the decoder.

[0094] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: using a subnetwork to obtain the potential value y or the quantized potential value The quantitative super potential value is obtained in.

[0095] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the second sub-network is autoregressive.

[0096] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: the second sub-network includes a context module.

[0097] Optionally, in any of the aforementioned aspects, another embodiment of this aspect provides: the second sub-network includes a super decoder module.

[0098] A third aspect relates to an apparatus for processing video data, comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform any of the disclosed methods.

[0099] A fourth aspect relates to a non-transitory computer-readable medium, comprising a computer program product for use by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, so that when the computer executable instructions are executed by a processor, the video codec device performs any of the disclosed methods.

[0100] A fifth aspect relates to a non-transitory computer-readable recording medium storing a bitstream of a video, wherein the bitstream is generated by a method performed by a video processing device, wherein the method includes any of the disclosed methods.

[0101] A sixth aspect relates to a method for storing a bitstream of a video, including any of the disclosed methods.

[0102] A seventh aspect relates to a method, apparatus or system described in this document.

[0103] For clarity, any of the aforementioned embodiments may be combined with any one or more of the other aforementioned embodiments to create new embodiments within the scope of the present disclosure.

[0104] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] For a more complete understanding of the present disclosure, reference is now made to the following description of the drawings taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0106] Figure 1 An example of a typical transform coding scheme is shown.

[0107] Figure 2 An example of quantization potential values ​​when using a super encoder / decoder is shown.

[0108] Figure 3 An example of a network architecture for an autoencoder implementing a hyper-prior model is shown.

[0109] Figure 4 A combined model is shown that jointly optimizes an autoregressive component that estimates the probability distribution of latent values ​​according to their causal context (i.e., the context model) as well as the hyper-prior and the underlying autoencoder.

[0110] Figure 5 The encoding process using a super encoder and a super decoder is shown.

[0111] Figure 6 An example of the decoding process is shown.

[0112] Figure 7 The decoder network with the problem in the dashed box is shown.

[0113] Figure 8 The entropy encoding and decoding subnetwork in the prior art image decoding architecture is shown.

[0114] Fig. 9 is an example of a decoding process according to an embodiment of the present disclosure.

[0115] Fig.10 is an example of an encoding process according to an embodiment of the present disclosure.

[0116] Fig.11 is an example of an encoding process according to an embodiment of the present disclosure.

[0117] Fig.12 is an example of an encoding process according to an embodiment of the present disclosure.

[0118] Fig.13 is an example of a context submodule.

[0119] Fig.14 is an example of a context submodule.

[0120] Fig.15 is an example of a context submodule.

[0121] Fig.16 is an example of a parallel convolution kernel.

[0122] Fig.17 is an example of a general implementation of the present disclosure.

[0123] Fig.18 is an example of a parallel convolution kernel.

[0124] Fig.19 is another example embodiment of the present disclosure.

[0125] Fig. 20 is another example of an embodiment of the present disclosure.

[0126] Fig.21 is a block diagram illustrating an example video processing system.

[0127] Fig. 22 is a block diagram of an example video processing device.

[0128] Fig.23 is a flow chart of an example method of video processing.

[0129] Fig.24 is a block diagram illustrating an example video codec system.

[0130] Fig.25 is a block diagram illustrating an example encoder.

[0131] Fig.26 is a block diagram illustrating an example decoder.

[0132] Fig. 27 is a schematic diagram of an example encoder.

[0133] Fig.28 It is an image or video method implemented by a decoding device according to an embodiment of the present disclosure.

[0134] Fig.29 It is an image or video method implemented by an encoding device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0135] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or yet to be developed. The present disclosure should in no way be limited to the illustrative implementations, drawings, and techniques shown below (including the exemplary designs and implementations illustrated and described herein), but may be modified within the scope of the appended claims and their full equivalents.

[0136] The section headings used in this document are intended to facilitate understanding, and do not limit the applicability of the techniques and embodiments disclosed in each section to only that section. In addition, the techniques described herein can be applied to other video codec protocols and designs.

[0137] 1. Summary.

[0138] A neural network-based image and video compression method includes an autoregressive subnetwork and an entropy coding engine, wherein entropy coding and decoding are performed independently of the autoregressive subnetwork.

[0139] 1. Background.

[0140] In the past decade, deep learning has been rapidly developed in various fields, especially in computer vision and image processing. Inspired by the great success of deep learning techniques in computer vision, many researchers have shifted their attention from traditional image / video compression techniques to neural image / video compression techniques. Neural networks were originally invented in the interdisciplinary research of neuroscience and mathematics. They show strong capabilities in the context of nonlinear transformation and classification. In the past five years, image / video compression techniques based on neural networks have made remarkable progress. It is reported that the latest neural network-based image compression algorithms [1] achieve comparable rate-distortion (RD) performance with Versatile Video Codec (VVC) [2], which is the latest video codec standard developed by experts from the Joint Video Experts Group (JVET) together with the Moving Picture Experts Group (MPEG) and the Video Coding Experts Group (VCEG). With the continuous improvement of neural image compression performance, neural network-based video compression has become an actively developing research field. However, due to the inherent difficulty of the problem, neural network-based video codecs are still in their early stages.

[0141] 2.1 Image / video compression.

[0142] Image / video compression generally refers to computing techniques that compress images / videos into binary codes for easy storage and transmission. The binary codes may or may not support lossless reconstruction of the original image / video, which is called lossless compression and lossy compression. Since lossless reconstruction is not necessary in most scenarios, most work is devoted to lossy compression. Usually, the performance of image / video compression algorithms is evaluated from two aspects, namely compression rate and reconstruction quality. The compression rate is directly related to the number of binary codes. The smaller the number of binary codes, the better the compression effect; the reconstruction quality is measured by comparing the reconstructed image / video with the original image / video. The higher the similarity, the better the reconstruction quality.

[0143] Image / video compression technology can be divided into two branches, classical video codec methods and neural network-based video compression methods. Classical video codec schemes adopt transform-based solutions, in which researchers exploit statistical dependencies in latent variables (e.g., discrete cosine transform (DCT) or wavelet coefficients) by manually designing entropy codes to simulate dependencies in quantization mechanisms. There are two forms of neural network-based video compression, namely neural network-based codec tools and end-to-end neural network-based video compression. The former is embedded into existing classical video codecs as a codec tool and is only part of the framework, while the latter is an independent framework developed based on neural networks and does not rely on classical video codecs.

[0144] Over the past three decades, a series of classic video codec standards have been developed to adapt to the increasing visual content. The International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) has two expert groups, the Joint Photographic Experts Group (JPEG) and the Moving Picture Experts Group (MPEG), and the International Telecommunication Union-Telecommunication Standardization Sector (ITU-T) also has its own Video Codec Experts Group (VCEG), which is responsible for the standardization of image / video codec technologies. Influential video codec standards released by these organizations include Joint Photographic Experts Group (JPEG), JPEG 2000, H.262, H.264 / Advanced Video Codec (AVC), and H.265 / High Efficiency Video Codec (HEVC). After H.265 / HEVC, the Joint Video Experts Group (JVET) composed of MPEG and VCEG has been working on a new video codec standard - Versatile Video Codec (VVC). The first version of VVC was released in July 2020. It is reported that VVC reduces the bit rate by 50% on average at the same visual quality compared to HEVC.

[0145] Neural network-based image / video compression is not a new technology, as many researchers have devoted themselves to neural network-based image encoding and decoding [3]. However, the network architecture is relatively shallow and the performance is not ideal. Benefiting from the support of abundant data and powerful computing resources, neural network-based methods have been better developed in various applications. Currently, neural network-based image / video compression has shown promising improvements, confirming its feasibility. However, the technology is still far from mature and many challenges need to be addressed.

[0146] 2.2 Neural Network.

[0147] Neural networks, also known as artificial neural networks (ANNs), are computational models used in machine learning techniques that typically consist of multiple processing layers, with each layer consisting of multiple simple but nonlinear basic computational units. One advantage of such deep networks is believed to be the ability to process data with multiple levels of abstraction and transform the data into different types of representations. Note that these representations are not manually designed; rather, the deep networks comprising the processing layers are learned from massive amounts of data using general machine learning procedures. Deep learning eliminates the need for handcrafted representations and is therefore believed to be particularly useful for processing natural unstructured data such as acoustic and visual signals, which has been a long-standing problem in the field of artificial intelligence.

[0148] 2.3 Neural Networks for Image Compression.

[0149] Existing neural networks for image compression methods can be divided into two categories, namely pixel probability models and autoencoders. The former belongs to predictive encoding and decoding strategies, while the latter is a transformation-based solution. Sometimes, these two methods are combined in the literature.

[0150] 2.3.1 Pixel probability model.

[0151] According to Shannon's information theory [6], the best approach for lossless codecs can achieve a minimum codec rate of -log2p(z), where p(z) is the probability of symbol x. Many lossless codecs have been proposed in the literature, among which arithmetic coding is considered to be one of the best methods [7]. Given a probability distribution p(x), arithmetic coding ensures that the codec rate is as close to its theoretical limit -log2p(x) as possible without considering round-off errors. Therefore, the remaining question is how to determine the probability, which is very challenging for natural images / videos due to the curse of dimensionality.

[0152] Following the predictive coding strategy, one way to model p(x) is to predict the pixel probability one by one in raster scan order based on previous observations, where x is the image.

[0153] p(x)=p(x1)p(x2|x1)…p(x i |x1,…,x i-1 )…p(x m×n |x1,…,x m×n-1 ) (1)

[0154] Where m and n are the height and width of the image, respectively. The previous observation is also called the context of the current pixel. When the image is large, it may be difficult to estimate the conditional probability. Therefore, a simplified approach is to limit the scope of its context.

[0155] p(x)=p(x1)p(x2|x1)…p(x i |x i-k ,…,x i-1 )…p(x m×n |x m×n-k ,…,x m×n-1 ) (2)

[0156] Where k is a predefined constant that controls the scope of the context.

[0157] It should be noted that the condition may also consider the sample values ​​of other color components. For example, when encoding and decoding RGB color components, the R sample depends on the previously encoded and decoded pixels (including R / G / B samples), and the current G sample can be encoded and decoded based on the previously encoded and decoded pixels and the current R sample, and when encoding and decoding the current B sample, the previously encoded and decoded pixels and the current R and G samples can also be considered.

[0158] Most compression methods directly model the probability distribution in the pixel domain. Some researchers have also tried to model the probability distribution as a conditional probability distribution based on explicit or latent representations. In other words, we can estimate

[0159]

[0160] Where h is an additional condition, and p(x)=p(h)p(x|h), which means that modeling is divided into unconditional modeling and conditional modeling. The additional condition can be image label information or high-level representation.

[0161] 2.3.2 Autoencoder.

[0162] Autoencoders are derived from the well-known work proposed by Hinton and Salakhutdinov

[17] . The method is trained for dimensionality reduction and consists of two parts: encoding and decoding. The encoding part converts the high-dimensional input signal into a low-dimensional representation, which usually has a reduced spatial size but a larger number of channels. The decoding part tries to recover the high-dimensional input from the low-dimensional representation. Autoencoders enable automatic learning of representations without the need for handcrafted features, which is considered one of the most important advantages of neural networks.

[0163] Figure 1 A typical transform coding scheme is shown. a The original image x is transformed to obtain the latent representation y. The latent representation y is quantized and compressed into bits. The number of bits R is used to measure the encoding and decoding rate. Then it is passed through the synthesis network g s Quantizing the Latent Representation Perform inverse transform to obtain the reconstructed image By using the function g p Transform x and To compute the distortion in perceptual space.

[0164] It is intuitive to apply autoencoder networks to lossy image compression. We just need to encode the latent representation learned from the trained neural network. However, adapting autoencoders to image compression is non-trivial because the original autoencoders are not optimized for compression, so directly using the trained autoencoders is not efficient. In addition, there are other significant challenges. First, the low-dimensional representation should be quantized before being encoded, but quantization is not differentiable, which is required in backpropagation when training neural networks. Second, the goals in the compression scenario are different because both distortion and rate need to be considered. Estimating the rate is challenging. Third, practical image coding and decoding schemes need to support variable rates, scalability, encoding / decoding speed, and interoperability. To address these challenges, many researchers are actively contributing to this field.

[0165] Prototype autoencoders for image compression such as Figure 1 As shown, it can be regarded as a transform coding strategy. Using the analysis network y = g a (x) transforms the original image x, where y is the latent representation that will be quantized and decoded. The synthesis network will transform the quantized latent representation Perform inverse transform to obtain the reconstructed image The framework is trained using the rate-distortion loss function, which is where D is x and The distortion between them, R is expressed from the quantization is the rate calculated or estimated in , and λ is the Lagrange multiplier. It should be noted that D can be calculated in the pixel domain or the perception domain. All existing research works follow this prototype, and the difference may only be in the network structure or loss function.

[0166] In terms of network structure, recurrent neural networks (RNNs) and convolutional neural networks (CNNs) are the most widely used architectures. In the RNN-related category, Toderici et al.

[18] proposed a general framework for RNN-based variable rate image compression. They used binary quantization to generate codes and did not consider the rate during training. The framework does provide scalable encoding and decoding capabilities, where RNNs with convolutional and deconvolutional layers are reported to perform well. Toderici et al.

[19] subsequently proposed an improved version by upgrading the encoder with a pixel RNN-like neural network to compress binary codes. It is reported that its performance is better than JPEG on the Kodak image dataset when using the multi-scale structural similarity (MS-SSIM) evaluation metric. Johnston et al.

[20] further improved the RNN-based solution by introducing a hidden state activation mechanism. In addition, an SSIM weighted loss function was designed and a spatially adaptive bitrate mechanism was enabled. They achieved better results than Better Portable Graphics (BPG) on the Kodak image dataset when using MS-SSIM as the evaluation metric. Covell et al.

[21] supported spatially adaptive bitrate by training a blocking-tolerant RNN.

[0167] Ballé et al.

[22] proposed a general framework for rate-distortion optimized image compression. Integer codes are generated using multi-level quantization and the rate during training is considered, i.e., the loss is a joint rate-distortion cost, which can be the mean square error (MSE) or other. They add random uniform noise to stimulate quantization during training and use the differential entropy of the noise code as a proxy for the rate. They use GDN (generalized divisive normalization) as the network structure, which consists of a linear mapping followed by a nonlinear parameter normalization. The effectiveness of GDN in image coding and decoding was verified in

[23] . Ballé et al.

[24] then proposed an improved version in which they used 3 convolutional layers as the forward transform, each followed by a downsampling layer and a GDN layer. Therefore, they used 3 layers of inverse GDN to simulate the inverse transform, each followed by an upsampling layer and a convolution layer. In addition, an arithmetic coding method was designed to compress integer codes. It is reported that the performance on the Kodak dataset is better than JPEG and JPEG 2000 in terms of MSE. In addition, Ballé et al.

[25] improved the method by designing a variance hyper-prior in the autoencoder. They have a subnet h a The potential representation y is transformed into z = h a (y), and z will be quantized and sent as side information. s Implementing the inverse transform, this subnetwork attempts to quantize the auxiliary information Decoding to quantization The standard deviation of is further used during arithmetic coding and decoding of . On the Kodak image set, their method is slightly inferior to BPG in terms of PSNR. D.Minnen et al.

[26] further exploit the structure in the residual space by introducing an autoregressive model to estimate the standard deviation and mean. In the latest work

[27] , Z.Cheng et al. use a Gaussian mixture model to further remove redundancy in the residual. Using peak signal-to-noise ratio (PSNR) as the evaluation metric, the reported performance on the Kodak image set is comparable to VVC

[28] .

[0168] 2.3.3 Super-prior model.

[0169] In the transform codec approach to image compression, the encoder subnetwork (Section 2.3.2) uses a parametric analysis transform The image vector x is transformed into a latent representation y, which is then quantized to form because It is a discrete value, so it can be losslessly compressed using entropy coding and decoding technology (such as arithmetic coding and decoding) and transmitted as a bit sequence.

[0170] from Figure 2 It can be clearly seen from the center-left and center-right images that There is significant spatial correlation between the elements of . Notably, their scales (center-right image) appear to be spatially coupled. In

[25] , an additional set of random variables was introduced to capture spatial correlation and further reduce redundancy. In this case, the image compression network is Figure 3 shown.

[0171] exist Figure 3 In the figure, the left side of the model is the encoder g a and decoder g s (Explained in Section 2.3.2). The right side is used to obtain The additional super encoder h a Network and Super Decoder s In this architecture, the encoder takes the input image x and feeds it to g a , resulting in a response y with a spatially varying standard deviation. The response y is fed to h a , thus summarizing the distribution of the standard deviation of z. z is then quantified compressed and transmitted as auxiliary information. The encoder then uses the quantized vector to estimate σ, the spatial distribution of the standard deviation, and use it to compress and transmit the quantized image representation The decoder first recovers the compressed signal It then uses h s to obtain σ, which provides it with the correct probability estimate to successfully recover It will then Feed g s to obtain the reconstructed image.

[0172] When super encoders and super decoders are added to image compression networks, the quantization potential is reduced. of space redundancy. Figure 2 The rightmost image in the figure corresponds to the quantized potential values ​​when using the super encoder / decoder. Compared to the center-right image, the spatial redundancy is significantly reduced because the samples of the quantized potential values ​​are less correlated.

[0173] exist Figure 2, an image from the Kodak dataset is shown on the left; a visualization of the latent representation y of the image is shown on the left; the standard deviation σ of the latent values ​​is shown on the right; and the latent values ​​y after being processed by the super-prior (super-encoder and decoder) network are shown on the right.

[0174] Figure 3 The network architecture of the autoencoder implementing the super-prior model is shown. The image autoencoder network is shown on the left, and the right corresponds to the super-prior subnetwork. The analysis and synthesis transformations are denoted as g a and g a Q represents quantization, and AE and AD represent arithmetic encoder and arithmetic decoder respectively. The super prior model consists of two sub-networks, the super encoder (with h a denoted) and super decoder (with h s denoted by ). The hyper-prior model generates a quantitative hyper-potential value It includes information about quantifying potential values The probability distribution information of the sample points. is included in the bitstream and is related to are transmitted together to the receiver (decoder).

[0175] 2.3.4 Context model.

[0176] Although the hyper-prior model improves the quantitative potential value , but additional improvements can be obtained by using autoregressive models to predict quantitative potential values ​​from their causal context (Context Model).

[0177] The term "autoregressive" means that the output of a process is later used as input to that process. For example, the context model subnetwork generates a sample of potential values, which is later used as input to get the next sample.

[0178]

[26] used a joint architecture, where both the super-prior model sub-network (super-encoder and super-decoder) and the context model sub-network were utilized. The super-prior and context models were combined to learn to quantize the latent value The probability model of is then used for entropy coding. Figure 4 As shown, the outputs of the context subnetwork and the super decoder subnetwork are combined by a subnetwork called the entropy parameter, which generates the mean μ and scale (or variance) σ parameters for the Gaussian probability model. The Gaussian probability model is then used to encode the samples of the quantized potential values ​​into a bitstream with the help of the arithmetic encoder (AE) module. In the decoder, the Gaussian probability model is used by the arithmetic decoder (AD) module to obtain the quantized potential values ​​from the bitstream

[0179] Figure 4 A combined model is shown that jointly optimizes an autoregressive component that estimates the probability distribution of latent values ​​according to their causal context (context model) as well as the hyperprior and the underlying autoencoder. The real-valued latent representation is quantized (Q) to create a quantized latent value and quantify the super potential value It is compressed into a bitstream via an arithmetic encoder (AE) and decompressed by an arithmetic decoder (AD). The highlighted areas correspond to the components executed by the receiver (ie, decoder) to restore the image from the compressed bitstream.

[0180] Typically, the latent samples are modeled as Gaussian distributions or Gaussian mixture models (but not limited to these). In

[26] and according to Figure 4 , the context model and the hyper-prior are jointly used to estimate the probability distribution of the potential sample points. Since the Gaussian distribution can be defined by the mean and variance (also known as sigma or scale), the joint model is used to estimate the mean and variance (denoted as μ and σ).

[0181] 2.3.5 Encoding process using the joint autoregressive hyper-prior model.

[0182] Figure 4 This corresponds to the prior art compression method proposed in

[26] . In this section and the next, the encoding and decoding processes are described respectively.

[0183] Figure 5 The encoding process according to

[26] is shown.

[0184] exist Figure 5 In the previous section, the encoding process is described. The input image is first processed by the encoder subnetwork. The encoder transforms the input image into a transformed representation called a potential value (denoted by y). y is then input to the quantizer block, denoted by Q, to obtain a quantized potential value Then it is converted into a bit stream (bits1) using an arithmetic coding module (denoted as AE). The arithmetic coding block sequentially converts Each sample point is converted into a bit stream (bits1) one by one.

[0185] The modular super encoder, context, super decoder, and entropy parameter subnetworks are used to estimate the quantized latent value. The potential value y is input to the super encoder, and the super encoder outputs the super potential value (represented by z). Then, the super potential value is quantized The factorization entropy module generates a probability distribution which is used to encode the quantized super potential value into a bitstream. The quantized super potential value includes information about the quantized potential value. information about the probability distribution of .

[0186] The entropy parameter subnetwork generates probability distribution estimates, which are used to encode quantized latent values. The information generated by the entropy parameters usually includes the mean μ and the scale (or variance) σ parameters, which are used together to obtain a Gaussian probability distribution. The Gaussian distribution of a random variable x is defined as where the parameter μ is the mean or expectation of the distribution (as well as its median and mode), and the parameter σ is its standard deviation (or variance, or scale). To define a Gaussian distribution, the mean and variance need to be determined. In

[26] , the entropy parameter module is used to estimate the mean and variance values.

[0187] The subnetwork superdecoder generates part of the information used by the entropy parameter subnetwork, and another part of the information is generated by an autoregressive module called the context module. The context module uses the samples that have been encoded by the arithmetic coding (AE) module to generate information about the probability distribution of the samples of the quantized potential value. It is usually a matrix composed of many sample points. Dimensions can be used such as or The index of the sample point is used to indicate the sample point. Encoded one by one by the AE, usually using raster scan order. In raster scan order, the rows of the matrix are processed from top to bottom, where the samples in the row are processed from left to right. In this scenario (where the AE encodes the samples into the bitstream using raster scan order), the context module uses the previously encoded samples in raster scan order to generate the same samples as the The information generated by the context module and the super decoder is combined by the entropy parameter module to generate a parameter for quantizing the potential value Encoded as a probability distribution of the bit stream (bits1).

[0188] Finally, the first and second bit streams are transmitted to a decoder as a result of the encoding process.

[0189] Note that other names may be used for the above modules.

[0190] In the above description, Figure 5 All elements in are collectively referred to as encoders. The analysis transformation that transforms the input image into a latent representation is also called an encoder (or autoencoder).

[0191] 2.3.6. Decoding process using the joint autoregressive hyper-prior model.

[0192] Figure 6The decoding process corresponding to

[26] is shown. Figure 6 The decoding process corresponding to

[26] is depicted separately.

[0193] During the decoding process, the decoder first receives the first bitstream (bits1) and the second bitstream (bits2) generated by the corresponding encoder. Bits2 is first decoded by the arithmetic decoding (AD) module using the probability distribution generated by the factorization entropy subnetwork. The factorization entropy module typically generates the probability distribution using a predetermined template, such as a predetermined mean and variance value in the case of a Gaussian distribution. The output of the arithmetic decoding process for bits2 is It is the quantized super potential value. The AD process is restored to the AE process applied in the encoder. The AE and AD processes are lossless, which means that the quantized super potential value generated by the encoder is Can be reconstructed at the decoder without any changes.

[0194] In obtaining After that, it is processed by the super decoder, whose output is fed to the entropy parameter module. The three sub-networks, context, super decoder, and entropy parameters employed in the decoder are the same as those in the encoder. Therefore, the exact same probability distribution is available in the decoder as in the encoder, which is essential for reconstructing the quantized latent values ​​without any loss. is crucial. Therefore, the quantized potential value obtained in the encoder can be obtained in the decoder The same version of .

[0195] After obtaining the probability distribution (e.g., mean and variance parameters) through the entropy parameter subnetwork, the arithmetic decoding module decodes the samples of the quantized potential value one by one from the bitstream bits1. From a practical point of view, the autoregressive model (context model) is inherently serial and therefore cannot be accelerated using techniques such as parallelization.

[0196] Finally, the fully reconstructed quantized potential value is input to the synthetic transform (in Figure 6 The decoder module is represented as a decoder in the figure to obtain the reconstructed image.

[0197] In the above description, Figure 6 All elements in are collectively referred to as decoders. The synthetic transform that transforms the quantized latent values ​​into the reconstructed image is also called a decoder (or autodecoder).

[0198] 2.4 Neural Networks for Video Compression.

[0199] Similar to traditional video codec technology, in neural network-based video compression, neural image compression serves as the basis for intra-frame compression, so the development of neural network-based video compression technology is later than that of neural network-based image compression technology, but due to its complexity, more efforts are needed to solve these challenges. Since 2017, some researchers have been working on neural network-based video compression schemes. Compared with image compression, video compression requires effective methods to eliminate inter-frame redundancy. Therefore, inter-frame prediction is a key step in these works. Motion estimation and compensation are widely adopted, but only recently have they been implemented by trained neural networks.

[0200] Research on neural network-based video compression can be divided into two categories according to the target scenario: random access and low latency. In the case of random access, it requires that decoding can start from any point in the sequence, which usually divides the entire sequence into multiple separate segments, and each segment can be decoded independently. In the case of low latency, its purpose is to reduce the decoding time, so that usually only the previous frame in time can be used as a reference frame to decode the subsequent frame.

[0201] 2.4.1 Low latency.

[0202]

[29] first proposed a video compression scheme using a trained neural network. They first divided the video sequence frames into blocks, and each block will choose one of the two available modes, intra-frame codec or inter-frame codec. When intra-frame codec is selected, there is an associated autoencoder to compress the block. When inter-frame codec is selected, motion estimation and compensation are performed using traditional methods, and the trained neural network will be used for residual compression. The output of the autoencoder is directly quantized and encoded by the Huffman method.

[0203] Chen et al.

[31] proposed another neural network-based video codec scheme using PixelMotionCNN. Frames are compressed in temporal order, and each frame is divided into blocks that are compressed in raster scan order. Each frame is first extrapolated using the two previous reconstructed frames. When a block is to be compressed, the extrapolated frame is fed into PixelMotionCNN along with the context of the current block to derive a latent representation. The residual is then compressed using a variable rate image scheme

[34] . The performance of this scheme is comparable to that of H.264.

[0204] Lu et al.

[32] proposed a realistic end-to-end neural network-based video compression framework, in which all modules are implemented using neural networks. The scheme accepts the current frame and the previously reconstructed frame as input, and uses the optical flow derived from a pre-trained neural network as motion information. The motion information is warped with the reference frame, and then the motion compensated frame is generated by the neural network. The residual and motion information are compressed using two separate neural autoencoders. The entire framework is trained using a single rate-distortion loss function. It achieves better performance than H.264.

[0205] Rippel et al.

[33] proposed an advanced video compression scheme based on neural networks. It inherits and extends the traditional neural network video coding scheme and has the following main features: 1) only one autoencoder is used to compress motion information and residuals; 2) motion compensation with multiple frames and multiple optical flows; 3) online states are learned and propagated through subsequent frames over time. The scheme has better performance than the HEVC reference software in terms of multi-scale structural similarity (MS-SSIM).

[0206] J.Lin et al.

[36] proposed an extended video compression framework based on end-to-end neural network based on

[32] . In this solution, multiple frames are used as references. Thus, it is possible to provide a more accurate prediction of the current frame by using multiple reference frames and associated motion information. In addition, motion field prediction is deployed to eliminate motion redundancy along the time domain channel. A post-processing network is also introduced in this study to remove reconstruction artifacts in the previous process. In terms of peak signal-to-noise ratio (PSNR) and MS-SSIM, the performance is significantly better than

[32] and H.265.

[0207] Eirikur et al.

[37] proposed a scale-space flow to replace the commonly used optical flow by adding a scale parameter based on the framework of

[32] . It is reported that it achieves better performance than H.264.

[0208] Z.Hu et al.

[38] proposed a multi-resolution representation for optical flow based on

[32] . Specifically, the motion estimation network generates multiple optical flows with different resolutions and lets the network learn which one to choose under the loss function. Compared with

[32] , the performance is slightly improved and better than H.265.

[0209] 2.4.2 Random access.

[0210] Wu et al.

[30] proposed a neural network-based video compression scheme with frame interpolation. The key frames are first compressed using a neural image compressor, and the remaining frames are compressed in a hierarchical order. They perform motion compensation in the perceptual domain, i.e., feature maps are derived at multiple spatial scales of the original frame and the motion is used to warp these feature maps, which are then used in the image compressor. The method is reported to be comparable to H.264.

[0211] Djelouah et al.

[41] proposed an interpolation-based video compression method, in which the interpolation model combines motion information compression and image synthesis, and the same autoencoder is used for both the image and the residual.

[0212] Amirhossein et al.

[35] proposed a neural network-based video compression method based on a variational autoencoder with a deterministic encoder. Specifically, the model includes an autoencoder and an autoregressive prior. Unlike previous methods, this method accepts group of pictures (GOP) as input and incorporates a three-dimensional (3D) autoregressive prior by considering temporal correlations when encoding and decoding the latent representation. It provides performance comparable to H.265.

[0213] 2.5 Introduction.

[0214] Almost all natural images / videos are in digital format. A grayscale digital image can be represented as in is a set of pixel values, m is the image height, and n is the image width. For example, is the normal setting, in which case Thus a pixel can be represented by an 8-bit integer. An uncompressed grayscale digital image has 8 bits per pixel (bpp), while compressed bits are certainly less.

[0215] Color images are usually represented in multiple channels to record color information. For example, in the RGB color space, an image can be represented by Representation, with three independent channels storing red, green, and blue information. Similar to an 8-bit grayscale image, an uncompressed 8-bit RGB image has 24bpp. Digital images / videos can be represented in different color spaces. Neural network-based video compression schemes have been developed primarily in the RGB color space, while traditional codecs typically use the YUV color space to represent video sequences. In the YUV color space, the image is decomposed into three channels, namely Y, Cb, and Cr, where Y is the luminance component and Cb / Cr are the chrominance components. Since the human visual system is less sensitive to the chrominance components, the benefit of this is that Cb and Cr are usually downsampled for pre-compression.

[0216] A color video sequence consists of multiple color images called frames to record scenes at different time stamps. For example, in the RGB color space, a color video can be composed of X = {x0, x1, ..., x t ,…,x T-1}, where T is the number of frames in the video sequence, If m = 1080, n = 1920, And the video has 50 frames per second (fps), then the data rate of the uncompressed video is 1920×1080×8×3×50=2,488,320,000 bits per second (bps), about 2.32 Gbps, which requires a lot of storage space, so it definitely needs to be compressed before transmission over the Internet.

[0217] Typically, for natural images, lossless methods can achieve a compression ratio of about 1.5 to 3, which is significantly lower than required. Therefore, lossy compression came into being to achieve a higher compression ratio, but at the expense of distortion. Distortion can be measured by calculating the average squared difference between the original image and the reconstructed image, i.e., the mean square error (MSE). For grayscale images, MSE can be calculated using the following equation.

[0218]

[0219] Therefore, the quality of the reconstructed image compared to the original image can be measured by the Peak Signal-to-Noise Ratio (PSNR):

[0220]

[0221] in yes The maximum value in , for example, for 8-bit grayscale images, the maximum value is 255. There are also other quality assessment metrics, such as structural similarity (SSIM) and multi-scale SSIM (MS-SSIM) [4].

[0222] To compare different lossless compression schemes, one only needs to compare the compression ratio for a given resulting rate, and vice versa. However, to compare different lossy compression methods, both the rate and the reconstruction quality must be considered. For example, a commonly used approach is to compute relative rates at several different quality levels and then average these computed rates; the average relative rate is called Bjontegaard's delta rate (BD-rate) [5]. There are other important aspects to evaluate image / video codec schemes, including encoding / decoding complexity, scalability, robustness, etc.

[0223] 3. Question

[0224] 3.1 Core Issues

[0225] Image compression networks in the prior art include autoregressive models (e.g., context models) to improve compression performance. However, the autoregressive model is interleaved with an entropy decoding process that is inherently serial, resulting in a decoding process that is essentially serial (not efficiently parallelizable) and very slow.

[0226] This problem relates to the entropy decoding part of the image decoding architecture in the prior art. Figure 7 The prior art decoder design is depicted. On the right, the module within the dashed rectangle is responsible for entropy decoding the quantized latent values Since its serial nature, this part is very slow in the prior art architecture.

[0227] 3.2 Problem details.

[0228] Figure 8 The entropy coding / decoding sub-network in the prior art image decoding architecture is shown.

[0229] In the prior art image decoding architecture according to

[26] , the process of reconstructing the quantized latent values is performed as follows.

[0230] 1. The quantized hyper-latent values are processed by the hyper-decoder to generate the first part of information. The first part of information is fed into the entropy parameter module.

[0231] 2. The following operations are performed recursively and serially to reconstruct the samples of the quantized latent values :

[0232] a. The context module uses the samples to generate the second part of information, where:

[0233] i. n < j, or

[0234] ii. m < i, if n equals j.

[0235] (The sample located at [m, n] is the already reconstructed sample).

[0236] b. Using the first and second parts of information, the entropy parameter module generates μ[i, j] and σ[i, j], which are the mean and variance of the Gaussian probability distribution, respectively.

[0237] c. The arithmetic decoder decodes the samples from the bitstream using the probability distribution whose mean and variance are μ[i, j] and σ[i, j], respectively In reconstructing the quantized latent values according to the above flowchart

[0238] After that, it is processed by the composite transform (decoder) to obtain the reconstructed picture. According to the notation used in

[26] , the composite transform is called the decoder. The above whole process (including Reconstruction of the image and reconstruction of the image) is also called decoding or decoder.

[0239] In the above, the sample points of the quantized potential value are given by Note that a sample point is not necessarily a scalar value, it may be a vector and may contain multiple elements. In the rest of this application, a sample point may be represented by or In the latter, ":" is used to indicate the presence of a third dimension and to emphasize that the sample point has multiple elements.

[0240] In reconstructing the quantified potential value After the samples of , a synthesis transform (ie, decoder) is performed to obtain the reconstructed image.

[0241] As is apparent from the above description, the arithmetic decoding operation and the context module operation form a decoding This means that it cannot be reconstructed in parallel. of samples, they need to be reconstructed one by one.

[0242] The arithmetic decoding process (not limited to arithmetic coding, including most other entropy coding methods such as range coding) is a computationally simple but essentially sequential operation. The reason is that the bit stream consists of a series of bits, and the bits need to be decoded one by one. The process is suitable for being performed by a fast processing unit such as a central processing unit (CPU).

[0243] On the other hand, the context module and the entropy parameter module are computationally intensive and highly parallelizable operations. They are more suitable for execution by a massively parallel processing unit such as a graphics processing unit (GPU).

[0244] When the context module and the entropy parameter module are interleaved with arithmetic decoding, problems arise in the prior art image codec architecture. As described in the flowchart above, The decoding of a sample of requires the application of a context module, an entropy parameter module, and then an arithmetic decoding module. The context module and the entropy parameter module are deep neural networks, which means that they include a large number of operations. Arithmetic decoding is a relatively simple operation, however, it is completely serial. The fully serial arithmetic decoding operation interleaved with the complex "context" and "entropy parameter" operations significantly slows down the decoding process.

[0245] As a first example, if the processes of the context, arithmetic decoding, and entropy parameter modules are executed in a GPU, the following occurs.

[0246] 1. First, use arithmetic decoding to get samples of the quantized potential value. Because arithmetic decoding is completely serial, only a single core of the GPU is used. During this time, all other GPU cores (there may be thousands of cores in a GPU) will be idle waiting. In addition, each core of the GPU is slow because they are not designed to perform calculations quickly.

[0247] 2. Once the samples of the quantized potential values ​​are obtained, the context module and the entropy parameter module are executed using multiple cores of the GPU. Since the context module and the entropy parameter module are suitable for large-scale parallelization (and therefore suitable for execution in a GPU), this second step can be performed efficiently.

[0248] 3. Go to step 1 until all samples of the quantized potential value are decoded.

[0249] Understandably, the source of the slowdown when decoding is performed on the GPU is step 1.

[0250] As a second example, if the processes of the context, arithmetic decoding, and entropy parameter modules are executed on a CPU, the following occurs.

[0251] 1. First, arithmetic decoding is used to obtain samples of the quantized potential values. Because arithmetic decoding is completely serial, the CPU is very suitable and thus obtains the samples very quickly.

[0252] 2. Once the sample points of the quantized potential values ​​are obtained, the context module and the entropy parameter module are executed. However, the CPU is not suitable for performing a large number of operations, which only includes a few processing cores. Therefore, this step will be very slow.

[0253] 3. Go to step 1 until all samples of the potential value have been decoded.

[0254] When this process is executed on a CPU, a slowdown is caused by step 2 which is not suitable for execution on a CPU.

[0255] Finally, as a third example, consider performing part of the decoding on the CPU (arithmetic decoding) and part of the decoding on the GPU (context and entropy parameters). In this case, the following happens.

[0256] 1. First, arithmetic decoding is used to get samples of the potential values. Because arithmetic decoding is completely serial, the CPU is very suitable and thus gets the samples very quickly. During this time, the GPU remains idle.

[0257] 2. The obtained data is passed to the GPU.

[0258] 3. Once the potential value samples are obtained, the context and entropy parameter modules are executed using multiple cores of the GPU. Since the context and entropy parameter modules are suitable for massive parallelization (and therefore suitable for execution in the GPU), this step 2 can be performed efficiently. During this time, the CPU remains idle.

[0259] 4. The obtained data (mean and variance values) are sent to the CPU.

[0260] 5. Go to step 1 until all samples of the potential value have been decoded.

[0261] Understandably, using the CPU and GPU in cascade is also not feasible due to the idle time of the CPU and GPU and the time required to transfer data back and forth between the CPU and GPU. In fact, this is the slowest implementation of the 3 options.

[0262] In short, a problem arises when computationally complex but massively parallelizable processes (such as deep neural networks, e.g. context modules or entropy parameter modules) need to be performed in an interleaved manner with simple but completely serial entropy coding processes (such as arithmetic coding or range coding). This problem exists in prior art image coding networks such as

[26] or [1], and thus the decoding process is very slow in this architecture.

[0263] A direct solution to the above problem is to remove the autoregressive "context" module, which eliminates the loop including the arithmetic decoder, context, and entropy parameter modules. However, this results in a significant reduction in compression efficiency. The present disclosure achieves the same results as the direct solution without sacrificing compression efficiency.

[0264] 4. The present disclosure.

[0265] The detailed techniques in this article should be considered as examples to explain the general concept. These techniques should not be interpreted in a narrow sense. In addition, these techniques can be combined in any way.

[0266] 4.1 Objectives of the present disclosure.

[0267] The goal of the present disclosure is to deinterleave the arithmetic decoding process and the computationally complex deep neural network operations. In other words, the goal of the present disclosure is to decouple the arithmetic decoding process from the neural network-based module. Therefore, the arithmetic decoding process can be completed independently without the need for input from the neural network-based process. As a result, the decoding speed is significantly improved.

[0268] 4.2 Embodiments of the present disclosure.

[0269] 4.2.1 Decoding process

[0270] Fig. 9is an example of a decoding process according to an embodiment of the present disclosure.

[0271] According to the present disclosure, the decoding operation is performed as follows.

[0272] 1. First, the second subnetwork is used to estimate the probability parameters ( Fig. 9 In ).

[0273] 2. The probability parameters (e.g., variance) generated by the second network are used to generate quantized residual potential values ​​(denoted as in the figure) by performing an arithmetic decoding process ). The arithmetic decoder decodes the received bit stream based on the probability parameter and generates

[0274] 3. Loop through the following steps until you get All elements of.

[0275] a. The first subnetwork is used to use the already obtained The sample points are used to estimate the quantitative potential value The mean parameter of .

[0276] b. Using quantized residual potential value and the mean to obtain The next element of .

[0277] 4. After obtaining After all samples of , a synthesis transformation (such as Figure 7 The decoder module in ) is used to obtain the reconstructed image.

[0278] In the above Fig. 9 An exemplary embodiment of the present disclosure is depicted in FIG. 10 (Decoding Process). In this figure, the first subnetwork includes context, prediction, and optional super decoder modules. The second network includes a super prior variance decoder module. The quantized super potential value is Compared with the existing technology ( Figure 7 ), the arithmetic decoding process is removed from the loop consisting of arithmetic decoding, context and entropy parameters. In contrast, according to the present disclosure, the arithmetic decoding process is performed without using any input from the context and entropy parameter module, so it can be performed independently (it is deinterleaved). According to the present disclosure, the arithmetic decoding module has two inputs, namely the bitstream and the probability parameter (e.g., variance), which is the output of the super-prior variance decoder. The super-prior variance decoder uses the quantized super potential value To generate the probability parameters. The arithmetic decoding process generates the quantized residual potential value

[0279] After obtaining the residual latent value, a recursive prediction operation is performed to obtain the latent value Sample points of potential value Obtained through the following steps.

[0280] 1. The autoregressive context module is used to use the sample points The first input to the prediction module is generated by , where the (m, n) pair is the index of the sample point whose potential value has been obtained.

[0281] 2. Optionally, by using a super decoder and quantizing the super potential To obtain the second input of the prediction module.

[0282] 3. Using the first input and the second input, the prediction module generates the mean mean[:,i,j].

[0283] 4. The mean mean[:,i,j] and the quantized residual potential value Add together to get the potential value

[0284] 5. Repeat steps 1-4 for the next sample point.

[0285] Fig.10 is an example of an encoding process according to an embodiment of the present disclosure.

[0286] Fig.10 Another exemplary embodiment of the present disclosure is depicted. Fig. 9 In comparison, Fig.10 In , the same quantized super potential value is used as the input to the super decoder and super prior variance decoder modules. The rest of the operations are the same as described above.

[0287] 4.2.2 Encoding processing.

[0288] Fig.11 is an example of an encoding process according to an embodiment of the present disclosure.

[0289] According to the present disclosure, the encoding operation is performed as follows.

[0290] Initially, applications such as Figure 5 The analysis transform of the encoder in is used to obtain all samples of the potential value y.

[0291] 1. First, loop through the following steps until the quantized residual potential value is obtained All elements of.

[0292] a. The first subnetwork is used to estimate the potential value y using the already obtained samples of the quantized potential value y The mean parameter of .

[0293] b. Subtract the mean from y to obtain the residual w, which is quantized to obtain the quantized residual potential value

[0294] c. Add to the mean to get the quantitative potential value

[0295] 2. Secondly, the second sub-network is used to use the quantized super potential value to estimate probability parameters (e.g., variance).

[0296] 3. The entropy encoder module uses the probability parameters to encode elements of the quantized residual potential value into the bitstream.

[0297] exist Fig.11 An exemplary embodiment of the present disclosure (encoding process) is depicted in FIG. In this figure, the first subnetwork includes context, prediction, and optional super decoder modules. The second network includes a super prior variance decoder module. Figure 5 ), the arithmetic coding process is removed from the loop consisting of arithmetic coding, context and entropy parameters. In contrast, according to the present disclosure, the arithmetic coding process is performed without using any input from the context and entropy parameter modules, so it can be performed independently (it is deinterleaved). According to the present disclosure, the arithmetic coding module has two inputs, namely the quantized residual potential value and the probability parameter (e.g., variance) which is the output of the super-prior variance decoder. The arithmetic coding process uses a probability model with a mean of zero. The super-prior variance decoder uses the super-potential value The arithmetic coding process generates a bit stream which is transmitted to the decoder.

[0298] The quantized residual potential value is obtained according to the following recursive prediction operation Sample points.

[0299] 1. The autoregressive context module is used to use the sample points The first input to the prediction module is generated by , where the (m, n) pairs are the indices of the already obtained samples of the potential value.

[0300] 2. Optionally, by using super decoders and super potentials To obtain the second input of the prediction module.

[0301] 3. Using the first input and the second input, the prediction module generates the mean mean[:,i,j].

[0302] 4. Subtract the mean mean[:,i,j] from the latent value y[:,i,j] to obtain the residual latent value w[:,i,j].

[0303] 5. Quantify the residual potential value to obtain the quantized residual potential value

[0304] 6. is added to mean[:,i,j] to obtain the quantized potential value Repeat steps 1-5 for the next sample point.

[0305] Once the quantized residual potential value is obtained according to the above recursive process For all samples of The second sub-network (super-prior variance decoder) is used to estimate the probability parameters used in the entropy coding process.

[0306] Fig.12 is an example of an encoding process according to an embodiment of the present disclosure. Fig.12 Another exemplary embodiment of an encoder of the present disclosure is depicted. Fig.11 In comparison, Fig.11 In , the same quantized super potential value is used as the input to the super decoder and super prior variance decoder modules. The rest of the operations are the same as described above.

[0307] 4.2.3 Details of the context submodule and prediction fusion submodule.

[0308] In this disclosure, the context model is used (in Fig.10 , Fig.11 , Fig.12 ...depicted as “context” in the figure), which is used to quantify the potential sample points The output of the context model is input into the prediction fusion module.

[0309] exist Fig.13 An example of a context submodule is provided in , where the context submodule includes a convolutional layer with a convolution kernel size of (4×3), the number of input channels is 192, and the number of output channels is 384. The input of the context submodule is And the output is fed to the prediction fusion submodule.

[0310] Fig.14 The example in shows a context submodule, which is also in Fig.13 According to the present disclosure, the output of the context submodule and the prediction fusion submodule is μ, which is the mean (e.g., The context submodule and the prediction fusion submodule are jointly used to generate the prediction μ.

[0311] Since the output of the context prediction fusion pair is μ, consider the input The number of samples considered as input is proportional to the size of the context submodule convolution kernel and the number of input channels of the context submodule convolution kernel.

[0312] For example, in Fig.14 In the example, the number of kernel sizes is 4×3 (4 samples in the horizontal direction by 3 samples in the vertical direction). And the number of input channels is 192. Increasing the number of input channels or the kernel size (e.g., the product of the size in the horizontal direction and the size in the vertical direction) will increase the number of channels considered (used) for prediction. The number of subsequent samples.

[0313] In addition, increasing the number of output channels of the context submodule will increase the information obtained from the input samples, thereby improving the prediction performance. Fig.14 In , the number of output channels of the convolution kernel of the context submodule is 384. Increasing the number of output channels can better estimate (i.e., predict μ close to ).

[0314] Another example design of the context submodule is Fig.15 Here, the kernel size is represented by K×L, and the input / output channels are represented by A and B, respectively. As mentioned earlier, in order to improve the performance of the prediction, it is necessary to increase A, B, K, or L. On the other hand, the complexity of the context module is proportional to the product of these four numbers, A×B×K×L. Therefore, increasing any one of the numbers will result in a proportional increase in complexity.

[0315] In order to improve the prediction performance and limit the complexity, Fig.16 In the example, the context submodule is divided into 2 parallel convolution kernels. The convolution kernel size of the first convolution layer is 3×2, and the size of the second convolution layer is 4×3. For the two convolution layers, the number of input channels and the number of output channels are 192. The outputs of the two convolutions are connected by a "concat" block, whose output is 384 layers. The "concat" block stacks the two input channels on top of each other. The number of channels at the output of the concat block is the sum of the number of channels in the input of the concat block.

[0316] The proposed prediction network (consisting of the context submodule and the prediction fusion network) provides Fig.14 The design presented in is close to the predicted performance of . This is because of the following reasons.

[0317] The number of samples considered as input is the same in both designs, which is proportional to 4×3.

[0318] After the “concat” block, the number of input channels and output channels is the same (384).

[0319] The sample points that are closer to the sample point to be predicted in space are more relevant to the sample point to be predicted. Fig.16 middle, Fig.14 The convolutional layer in is divided into 2 convolution kernels, one of which is smaller (size 3×2). The convolution kernel with a smaller size helps to focus on the input samples that are spatially closer to the sample to be predicted. The larger convolution kernel 4×3 takes into account both the samples that are closer and farther away in space. Therefore, Fig.16 The entire design in can focus more on the samples near the sample to be predicted, while also considering the samples farther away (which are also correlated with the sample to be predicted).

[0320] Although the performance is very close, Fig.16 The complexity is reduced by 25%. Fig.16 The complexity of the design in is proportional to (4×3×192×192+3×2×192×192), while Fig.14 The complexity of the design in is proportional to 4×3×192×384. Therefore, the partition Fig.14 The convolutional layers in not only help maintain the prediction performance, but also have the additional effect of reducing the complexity.

[0321] exist Fig.17 A more general implementation of the present disclosure is depicted in . According to the present disclosure, the input potential sample is processed by at least two convolutional layers, and the convolution kernel sizes of the two convolutional layers are O×P and M×N, respectively, where the convolution kernel sizes are different in at least one direction (for example, O is not equal to M or P is not equal to N). The outputs of the two convolutional layers are connected and input to the prediction fusion network. The prediction fusion network outputs the predicted value μ as the estimate of the potential sample.

[0322] Fig.18 is an example of a parallel convolution kernel. Fig.18 Another embodiment of the present disclosure is depicted. According to the present disclosure, the input potential sample is processed by a first convolution layer with a convolution kernel size of M×N, and its output is processed by at least a second convolution layer (whose convolution kernel size is K×L in the figure). The output of the first convolution layer and the output of the second convolution layer are connected and then input to the prediction fusion module. The output of the prediction fusion submodule is μ, which is an estimate of the potential sample.

[0323] and Fig.17 Similar to the example of the present disclosure in Fig.18In the convolutional layer, the prediction performance is improved by keeping the complexity low. When two convolutional layers are applied successively, the receptive field of the second layer (i.e., the number of samples considered by the second convolutional layer) increases. This is because the first convolutional layer uses a region of input samples of size M×N to generate one output sample. The second convolutional layer uses the output of the first layer and considers a region of K×L samples. Therefore, the second layer effectively uses (K+M)×(L+N) input samples, which is why the receptive field of the second convolutional layer is increased.

[0324] Fig.18 The prediction performance of the design in is similar to that of a single convolutional layer with kernel (K+M)×(L+N). The complexity is proportional to (K×L+M×N) instead of (K+M)×(L+N), which is an advantage of this example implementation.

[0325] exist Fig.19 In the embodiment of the present disclosure, another exemplary embodiment of the present disclosure is presented. According to the present disclosure, potential sample points is fed to the first convolutional layer, and the residual latent samples is fed to the second convolutional layer. The output of the first convolutional layer and the output of the second convolutional layer are concatenated using a “concat” block and fed to the prediction fusion module. The output of the prediction fusion submodule is μ, which is an estimate of the potential sample point.

[0326] Fig.19 The advantage of the implementation in is that both the potential sample and the residual potential sample are related to the potential sample to be predicted. Therefore, both are used as inputs of the predictor to improve the prediction performance.

[0327] exist Fig. 20 , another example of an implementation is depicted. Fig.19 and Fig. 20 is the same, except that the output of the prediction fusion module is the potential sample to be predicted Rather than μ (which is an estimate of the potential sample point). Fig.19 In the above example, we need to combine the predicted μ with the residual sample point Add to get the potential sample points to be predicted. On the other hand, Fig. 20 In the process, the potential sample points to be predicted are directly obtained as the output of the prediction fusion module.

[0328] In the above examples of the present disclosure, the input potential samples and the output potential samples (of the context module and the prediction fusion module, respectively) sometimes use the same letter To express. It is usually a matrix or tensor containing multiple potential samples. Therefore, the input is the sample points that have been obtained through prediction. In other words, the prediction module (including the context module and the prediction fusion module) is an autoregressive neural network that processes the potential values ​​in a certain order. Sample points.

[0329] Additionally or alternatively, the "concat" block (i.e., the concatenation operation of the inputs) may be replaced by other fusion methods. The present disclosure is not limited to the concatenation of the outputs of at least two convolutional layers, and other methods for merging or fusing the outputs of the concatenated layers are also applicable to the present disclosure. One such fusion method may be addition.

[0330] 4.3. Differences between the prior art and the present disclosure.

[0331] There are at least three differences between the prior art and the present disclosure.

[0332] 1. Arithmetic coding and decoding are performed independently of the autoregressive subnetwork (first subnetwork). In this way, the fully sequential arithmetic coding / decoding process can be performed by a processing unit as fast as a CPU. A new subnetwork (second subnetwork) is introduced to estimate the probability parameters used by the arithmetic coding / decoding process.

[0333] 2. The arithmetic coding / decoding process is used to encode / decode the quantized latent residual, rather than encoding / decoding the quantized latent value as in the prior art.

[0334] 3. The autoregressive subnetwork is only used to estimate the mean of the potential value. In the prior art, it is used to estimate the mean and variance of the Gaussian distribution, which are then used by the arithmetic encoder and decoder to encode / decode samples of the quantized potential value.

[0335] 4.4 Beneficial effects of the present disclosure.

[0336] The beneficial effects of the present disclosure are as follows.

[0337] 1. The entropy decoding (e.g., arithmetic decoding) process can be performed independently, which is a simple but completely serial operation. For example, the entropy decoding process can be performed by a processing unit (such as a CPU) that is suitable for quickly performing serial operations. Once the entropy decoding operation is completed, the obtained data (quantized residual potential value) can be transmitted to the GPU.

[0338] 2. Computationally heavy but easily parallelizable modules (such as context and entropy parameter modules) can be performed independently of arithmetic coding / decoding. For example, a processing unit suitable for massively parallel processing (such as a GPU) can be used to perform these operations.

[0339] 3. Eliminates idle processing time.

[0340] According to the present disclosure, the CPU and GPU can be used in cascade. As an example, the decoding process can be performed as follows.

[0341] 1. First, the entropy decoding process is executed and completed. After the whole process is completed, all the samples of the quantized potential residual are obtained.

[0342] 2. Transfer the acquired data to the GPU. The data transfer happens only once.

[0343] 3. The context and entropy parameter modules are executed by the GPU and completed. No idle waiting occurs in the GPU. The quantized potential value is obtained.

[0344] 4. Perform synthesis transformation (decoder) to obtain the reconstructed image.

[0345] Compared with the third example in Section 3.2, the present disclosure eliminates the round-trip data transmission between the CPU and the GPU when both are used for decoding. In addition, it eliminates the idle waiting time.

[0346] In fact, the present disclosure can significantly increase the decoding speed by a factor of 10. Moreover, by clever design, it does not suffer any loss in compression efficiency.

[0347] 4.5 Examples of the Disclosure

[0348] The following detailed techniques should be considered as examples to explain the general concept. These techniques should not be interpreted in a narrow sense. In addition, these techniques can be combined in any way.

[0349] 1. In one example, more than one sub-network can be utilized as a super encoder / decoder for super information.

[0350] a. In one example, at least one sub-network is used to generate super information, which relies on a parsing process for latent information.

[0351] b. In one example, at least one sub-network is utilized to generate super-information that is independent of a parsing process for underlying information.

[0352] c. In one example, at least one sub-network is utilized to generate super information, which is used to predict potential signals.

[0353] d. In one example, the super information may include statistical information or probability distribution information of the underlying signal that may be quantified.

[0354] i. Statistical information or probability distribution information may include the mean of the underlying signal.

[0355] ii. Statistical information or probability distribution information may include the variance of the underlying signal.

[0356] 2. In one example, the underlying signal may be encoded and decoded in a predictive manner.

[0357] a. In one example, y'=y-p can be encoded in the encoder, where y is the potential sample and p is the prediction.

[0358] i. Correspondingly, y*=y'+p can be reconstructed in the decoder.

[0359] b. In one example, y may be quantized prior to the prediction process.

[0360] c. In one example, y may not be quantized before the prediction process.

[0361] d. In one example, p can be quantized before the prediction process.

[0362] e. In one example, p may not be quantized before the prediction process.

[0363] f. In one example, y' may be quantized after the prediction process.

[0364] g. In one example, y' may not be quantized after the prediction process.

[0365] h. In one example, at least one sub-network may be utilized to generate a prediction p.

[0366] i. In one example, at least one previously decoded y* or y' may be utilized to generate a prediction p for the current y or y*.

[0367] More details about the reference files can be found at:

[0368] [1] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 7939–7948, 2020.

[0369] [2] B. Bross, J. Chen, S. Liu, and Y.-K. Wang, “Versatile Video Draft (Draft 10),” JVET-S2001, July 2020.

[0370] [3] R. D. Dony and S. Haykin, “Neural network approaches to image compression,” Proceedings of the IEEE, vol. 83, no. 2, pp. 288–303, 1995.

[0371] [4] Z. Wang, A.C. Bovik, H.R. Sheikh and E.P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004.

[0372] [5] G. Bjontegaard, “Calculation of average PSNR differences between RD-curves,” VCEG, Tech. Rep. VCEG-M33, 2001.

[0373] [6] C.E. Shannon, “A mathematical theory of communication,” Bell System Technical Journal, vol. 27, no. 3, pp. 379–423, 1948.

[0374]

[17] G.E. Hinton and R.R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science, vol. 313, no. 5786, pp. 504–507, 2006.

[0375]

[18] G. Toderici, S. M. O’Malley, S. J. Hwang, D. Vincent, D. Minnen, S. Baluja, M. Covell, and R. Sukthankar, “Variable rate image compression with recurrent neural networks,” arXiv preprint arXiv:1511.06085, 2015.

[0376]

[19] G. Toderici, D. Vincent, N. Johnston, S. J. Hwang, D. Minnen, J. Shor, and M. Covell, “Full resolution image compression with recurrent neural networks,” in CVPR, 2017, pp. 5306–5314.

[0377]

[20] N. Johnston, D. Vincent, D. Minnen, M. Covell, S. Singh, T. Chinen, S. JinHwang, J. Shor, and G. Toderici, “Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks,” in CVPR, 2018, pp. 4385–4393.

[0378]

[21] M. Covell, N. Johnston, D. Minnen, S. J. Hwang, J. Shor, S. Singh, D. Vincent, and G. Toderici, “Target-quality image compression with recurrent,convolutional neural networks,” arXiv preprint arXiv:1705.06687, 2017.

[0379]

[22] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimization of nonlinear transform codes for perceptual quality,” in PCS. IEEE, 2016, pp. 1–5.

[0380]

[23] J. Ballé, “Efficient nonlinear transforms for lossy image compression,” in ′PCS, 2018, pp. 248–252.

[0381]

[24] J. Ballé, V. Laparra, and E. Simoncelli, “End-to-end optimized image compression,” in International Conference on Learning Representations, 2017.

[0382]

[25] J. Ballé, D. Minnen, S. Singh, S. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” in International Conference on Learning Representations, 2018.

[0383]

[26] D. Minnen, J. Ballé, G. Toderici, “Joint Autoregressive and Hierarchical Priors for Learned Image Compression,” arXiv.1809.02736.1, 2, 3, 4, 7.

[0384]

[27] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Learned image compression with discretized Gaussian mixture likelihoods and attention modules,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

[0385]

[28] Github repository “CompressAI:https: / / github.com / InterDigitalInc / CompressAI,” InterDigital Inc, accessed December 2020.

[0386]

[29] T. Chen, H. Liu, Q. Shen, T. Yue, X. Cao, and Z. Ma, “DeepCoder: A deep neural network based video compression,” in VCIP. IEEE, 2017, pp. 1–4.

[0387]

[30] C.-Y. Wu, N. Singhal, and P. Krahenbuhl, “Video compression through image interpolation,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 416–431.

[0388]

[31] Z. Chen, T. He, X. Jin, and F. Wu, “Learning for video compression,” IEEE Transactions on Circuits and Systems for Video Technology, DOI: 10.1109 / TCSVT.2019.2892608, 2019.

[0389]

[32] G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai, and Z. Gao, “DVC: ​​An end-to-end deep video compression framework,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

[0390]

[33] O. Rippel, S. Nair, C. Lew, S. Branson, A. Anderson, and L. Bourdev, “Learned Video Compression,” in 2019 IEEE / CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp. 3453–3462. doi:10.1109 / ICCV.2019.00355.

[0391]

[34] G. Toderici, D. Vincent, N. Johnston, S. J. Hwang, D. Minnen, J. Shor, and M. Covell, “Full resolution image compression with recurrent neural networks,” in CVPR, 2017, pp. 5306–5314.

[0392]

[35] A. Habibian, T. Rozendaal, J. Tomczak, and T. Cohen, “Video Compression with Rate-Distortion Autoencoders,” in Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), 2019, pp. 7033–7042.

[0393]

[36] J. Lin, D. Liu, H. Li, and F. Wu, “M-LVC: Multiple frames prediction for learned video compression,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

[0394]

[37] E. Agustsson, D. Minnen, N. Johnston, J. Ballé, S. J. Hwang, and G. Toderici, “Scale-Space Flow for End-to-End Optimized Video Compression,” in 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 8500–8509. doi:10.1109 / CVPR42600.2020.00853.

[0395]

[38] X. Hu, Z. Chen, D. Xu, G. Lu, W. Ouyang, and S. Gu, “Improving deep video compression by resolution-adaptive flow coding,” in European Conference on Computer Vision (ECCV) 2020.

[0396]

[41] Abdelaziz Djelouah, Joaquim Campos, Simone Schaub-Meyer, and Christopher Schroers. Neural inter-frame compression for video coding. In ICCV, pages 6421–6429, October 2019.

[0397] Fig.21 4000 is a block diagram illustrating an example video processing system 4000 in which the various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of system 4000. System 4000 may include an input 4002 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8 or 10 bit multi-component pixel values), or may be received in a compressed or encoded format. Input 4002 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, a passive optical network (PON), and wireless interfaces such as wireless fidelity (Wi-Fi) or a cellular interface.

[0398] System 4000 may include a codec component 4004, which may implement various codecs or encoding methods described in this document. Codec component 4004 may reduce the average bit rate of the video from input 4002 to the output of codec component 4004 to generate a codec representation of the video. Therefore, codec technology is sometimes referred to as video compression or video transcoding technology. The output of codec component 4004 may be stored or transmitted via the connected communication represented by component 4006. The stored or communicated bitstream (or codec) representation of the video received at input 4002 may be used by component 4008 to generate pixel values ​​or be sent to a displayable video of display interface 4010. The process of generating user-visible video from bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "codec" operations or tools, it should be understood that codec tools or operations are used at the encoder, and the corresponding decoding tools or operations of the result of the inverse codec will be performed by the decoder.

[0399] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB) or High Definition Multimedia Interface (HDMI) or DisplayPort, etc. Examples of storage interfaces include Serial Advanced Technology Attachment (SATA), Peripheral Component Interconnect (PCI), Integrated Drive Electronics (IDE) interface, etc. The techniques described in this document may be embodied in various electronic devices, such as mobile phones, laptop computers, smart phones, or other devices capable of performing digital data processing and / or video display.

[0400] Fig. 22 4100. The apparatus 4100 may be used to implement one or more methods described herein. The apparatus 4100 may be implemented in a smart phone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. The apparatus 4100 may include one or more processors 4102, one or more memories 4104, and a video processing circuit 4106. The processor(s) 4102 may be configured to implement one or more methods described in this document. The memory(s) 4104 may be used to store data and code for implementing the methods and techniques described herein. The video processing circuit 4106 may be used to implement some of the techniques described in this document in hardware circuits. In some embodiments, the video processing circuit 4106 may be at least partially included in the processor 4102, such as a graphics coprocessor.

[0401] Fig.234200. The method 4200 includes: at step 4202, determining to apply a pre-processing function to visual media data as part of an image compression framework. At step 4204, performing conversion between the visual media data and a bitstream based on the image compression framework. Depending on the example, the conversion of step 4204 may include encoding at an encoder or decoding at a decoder.

[0402] It should be noted that the method 4200 may be implemented in an apparatus for processing video data, the apparatus comprising a processor and a non-transitory memory having instructions thereon, such as the video encoder 4400, the video decoder 4500, and / or the encoder 4600. In this case, the instructions executed by the processor cause the processor to perform the method 4200. In addition, the method 4200 may be performed by a non-transitory computer-readable medium comprising a computer program product for use by a video codec device. The computer program product comprises computer-executable instructions stored on a non-transitory computer-readable medium, such that the computer-executable instructions, when executed by the processor, cause the video codec device to perform the method 4200.

[0403] Fig.24 43 is a block diagram illustrating an example video codec system 4300 that can utilize the techniques of the present disclosure. The video codec system 4300 may include a source device 4310 and a target device 4320. The source device 4310 generates encoded video data, which may be referred to as a video encoding device. The target device 4320 may decode the encoded video data generated by the source device 4310, which may be referred to as a video decoding device.

[0404] Source device 4310 may include video source 4312, video encoder 4314 and input / output (I / O) interface 4316. Video source 4312 may include sources such as video capture devices, interfaces for receiving video data from video content providers, and / or computer graphics systems for generating video data, or a combination of these sources. Video data may include one or more pictures. Video encoder 4314 encodes video data from video source 4312 to generate a bit stream. The bit stream may include a bit sequence that forms a codec representation of video data. The bit stream may include codec pictures and associated data. Codec pictures are codec representations of pictures. Associated data may include sequence parameter sets, picture parameter sets, and other syntax structures. I / O interface 4316 may include a modulator / demodulator (modem) and / or a transmitter. The encoded video data may be directly sent to target device 4320 via network 4330 via I / O interface 4316. The encoded video data may also be stored on storage medium / server 4340 for access by target device 4320.

[0405] The target device 4320 may include an I / O interface 4326, a video decoder 4324, and a display device 4322. The I / O interface 4326 may include a receiver and / or a modem. The I / O interface 4326 may obtain encoded video data from the source device 4310 or the storage medium / server 4340. The video decoder 4324 may decode the encoded video data. The display device 4322 may display the decoded video data to the user. The display device 4322 may be integrated with the target device 4320 or may be disposed outside the target device 4320, and the target device 4320 may be configured to be connected to an external display device.

[0406] The video encoder 4314 and the video decoder 4324 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other current and / or future standards.

[0407] Fig.25 is a block diagram showing an example of a video encoder 4400, which may be Figure 6 4300. The video encoder 4400 may be configured to perform any or all of the techniques of the present disclosure. The video encoder 4400 includes multiple functional components. The techniques described in the present disclosure may be shared between various components of the video encoder 4400. In some examples, the processor may be configured to perform any or all of the techniques described in the present disclosure.

[0408] The functional components of the video encoder 4400 may include a segmentation unit 4401, a prediction unit 4402 (which may include a mode selection unit 4403, a motion estimation unit 4404, a motion compensation unit 4405, an intra-frame prediction unit 4406), a residual generation unit 4407, a transform processing unit 4408, a quantization unit 4409, an inverse quantization unit 4410, an inverse transform unit 4411, a reconstruction unit 4412, a buffer 4413 and an entropy coding unit 4414.

[0409] In other examples, the video encoder 4400 may include more, fewer, or different functional components. In one example, the prediction unit 4402 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, in which at least one reference picture is a picture in which the current video block is located.

[0410] Furthermore, some components, such as the motion estimation unit 4404 and the motion compensation unit 4405 , may be highly integrated but are represented separately in the example of the video encoder 4400 for purposes of explanation.

[0411] The segmentation unit 4401 may segment the picture into one or more video blocks. The video encoder 4400 and the video decoder 4500 may support various video block sizes.

[0412] The mode selection unit 4403 may select one of the intra or inter coding modes, for example, based on the error result, and provide the resulting intra or inter coded block to the residual generation unit 4407 to generate residual block data, and to the reconstruction unit 4412 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 4403 may select a combined intra and inter prediction (CIIP) mode, in which the prediction is based on an inter prediction signal and an intra prediction signal. In the case of inter prediction, the mode selection unit 4403 may also select a resolution of motion vectors for the block (e.g., sub-pixel or integer pixel precision).

[0413] To perform inter-frame prediction on the current video block, the motion estimation unit 4404 may generate motion information for the current video block by comparing the current video block with one or more reference frames from the buffer 4413. The motion compensation unit 4405 may determine a predicted video block for the current video block based on the motion information and decoded samples of a picture from the buffer 4413 that is different from the picture associated with the current video block.

[0414] The motion estimation unit 4404 and the motion compensation unit 4405 may perform different operations on the current video block, for example, depending on whether the current video block is in an I slice, a P slice, or a B slice.

[0415] In some examples, the motion estimation unit 4404 may perform unidirectional prediction on the current video block, and the motion estimation unit 4404 may search the reference picture of list 0 or list 1 for the reference video block of the current video block. Then, the motion estimation unit 4404 may generate a reference index indicating the reference picture in list 0 or list 1 containing the reference video block and a motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 4404 may output the reference index, the prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 4405 may generate a predicted video block of the current block based on the reference video block indicated by the motion information of the current video block.

[0416] In other examples, the motion estimation unit 4404 may perform bidirectional prediction on the current video block, and the motion estimation unit 4404 may search for a reference video block of the current video block in the reference pictures in list 0, and may also search for another reference video block of the current video block in the reference pictures in list 1. Then, the motion estimation unit 4404 may generate a reference index indicating the reference pictures containing the reference video blocks in list 0 and list 1, and a motion vector indicating a spatial displacement between the reference video block and the current video block. The motion estimation unit 4404 may output the reference index and the motion vector of the current video block as the motion information of the current video block. The motion compensation unit 4405 may generate a predicted video block of the current video block based on the reference video block indicated by the motion information of the current video block.

[0417] In some examples, motion estimation unit 4404 may output a complete set of motion information for use in a decoding process by a decoder. In some examples, motion estimation unit 4404 may not output a complete set of motion information for the current video. Instead, motion estimation unit 4404 may signal the motion information of the current video block with reference to motion information of another video block. For example, motion estimation unit 4404 may determine that the motion information of the current video block is sufficiently similar to the motion information of a neighboring video block.

[0418] In one example, the motion estimation unit 4404 may indicate a value in a syntax structure associated with the current video block that indicates to the video decoder 4500 that the current video block has the same motion information as another video block.

[0419] In another example, the motion estimation unit 4404 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 4500 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0420] As discussed above, the video encoder 4400 can predictively signal motion vectors. Two examples of predictive signaling techniques that can be implemented by the video encoder 4400 include advanced motion vector prediction (AMVP) and merge mode signaling.

[0421] The intra prediction unit 4406 may perform intra prediction on the current video block. When the intra prediction unit 4406 performs intra prediction on the current video block, the intra prediction unit 4406 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a predicted video block and various syntax elements.

[0422] The residual generation unit 4407 may generate residual data of the current video block by subtracting the prediction video block(s) of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample components of the samples in the current video block.

[0423] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 4407 may not perform a subtraction operation.

[0424] The transform processing unit 4408 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0425] After the transform processing unit 4408 generates a transform coefficient video block associated with the current video block, the quantization unit 4409 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0426] The inverse quantization unit 4410 and the inverse transform unit 4411 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 4412 may add the reconstructed residual video block to corresponding samples from one or more predicted video blocks generated by the prediction unit 4402 to generate a reconstructed video block associated with the current block for storage in the buffer 4413.

[0427] After the reconstruction unit 4412 reconstructs the video block, a loop filtering operation may be performed to reduce video block artifacts in the video block.

[0428] The entropy coding unit 4414 may receive data from other functional components of the video encoder 4400. When the entropy coding unit 4414 receives data, the entropy coding unit 4414 may perform one or more entropy coding operations to generate entropy coded data and output a bitstream including the entropy coded data.

[0429] Fig.26 is a block diagram showing an example of a video decoder 4500, which may be Figure 6 Video decoder 4324 in the system 4300 shown. Video decoder 4500 can be configured to perform any or all of the techniques of the present disclosure. In the example shown, video decoder 4500 includes multiple functional components. The techniques described in this disclosure can be shared between various components of video decoder 4500. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0430] In the example shown, the video decoder 4500 includes an entropy decoding unit 4501, a motion compensation unit 4502, an intra prediction unit 4503, an inverse quantization unit 4504, an inverse transform unit 4505, a reconstruction unit 4506, and a buffer 4507. In some examples, the video decoder 4500 may perform a decoding process that is generally the inverse of the encoding process described with respect to the video encoder 4400.

[0431] The entropy decoding unit 4501 may retrieve a coded bitstream. The coded bitstream may include entropy-coded video data (e.g., coded blocks of video data). The entropy decoding unit 4501 may decode the entropy-coded video data, and the motion compensation unit 4502 may determine motion information including motion vectors, motion vector precision, reference picture list indexes, and other motion information from the entropy-decoded video data. For example, the motion compensation unit 4502 may determine such information by performing AMVP and merge modes.

[0432] The motion compensation unit 4502 may generate a motion compensated block, possibly performing interpolation based on an interpolation filter. An identifier of an interpolation filter used with sub-pixel precision may be included in a syntax element.

[0433] The motion compensation unit 4502 may calculate interpolation of sub-integer pixels of a reference block using an interpolation filter used by the video encoder 4400 during encoding of the video block. The motion compensation unit 4502 may determine the interpolation filter used by the video encoder 4400 based on received syntax information and use the interpolation filter to generate a prediction block.

[0434] The motion compensation unit 4502 can use some of the syntax information to determine: the size of the blocks used to encode (multiple) frames and / or (multiple) slices of the encoded video sequence, the partitioning information describing how each macroblock of the pictures of the encoded video sequence is partitioned, the mode indicating how to encode each partition, one or more reference frames (and reference frame lists) for each inter-frame codec block, and other information for decoding the encoded video sequence.

[0435] The intra prediction unit 4503 may use, for example, an intra prediction mode received in the bitstream to form a prediction block from spatially neighboring blocks. The inverse quantization unit 4504 inversely quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 4501. The inverse transform unit 4505 applies an inverse transform.

[0436] The reconstruction unit 4506 may sum the residual block with the corresponding prediction block generated by the motion compensation unit 4502 or the intra prediction unit 4503 to form a decoded block. If necessary, a deblocking filter may also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in a buffer 4507, which provides reference blocks for subsequent motion compensation / intra prediction and also produces a decoded video for presentation on a display device.

[0437] Fig. 27 4600 is a schematic diagram of an example encoder 4600. The encoder 4600 is suitable for implementing the techniques of VVC. The encoder 4600 includes three loop filters, namely a deblocking filter (DF) 4602, a sample adaptive offset (SAO) 4604, and an adaptive loop filter (ALF) 4606. Unlike the DF 4602, which uses a predetermined filter, the SAO 4604 and the ALF 4606 use the original samples of the current picture, and reduce the mean square error between the original samples and the reconstructed samples by adding an offset and applying a finite impulse response (FIR) filter, respectively, where the auxiliary information of the codec signals the offset and the filter coefficients. The ALF 4606 is located at the last processing stage of each picture and can be considered as a tool that attempts to capture and repair artifacts created by previous stages.

[0438] The encoder 4600 also includes an intra prediction component 4608 and a motion estimation / compensation (ME / MC) component 4610 configured to receive an input video. The intra prediction component 4608 is configured to perform intra prediction, while the ME / MC component 4610 is configured to perform inter prediction using a reference picture obtained from a reference picture buffer 4612. The residual block from the inter prediction or intra prediction is fed to a transform (T) component 4614 and a quantization (Q) component 4616 to generate a quantized residual transform coefficient, which is fed to an entropy coding component 4618. The entropy coding component 4618 entropy codes and decodes the prediction result and the quantized transform coefficient and transmits it to a video decoder (not shown). The quantization component output from the quantization component 4616 can be fed to an inverse quantization (IQ) component 4620, an inverse transform component 4622, and a reconstruction (REC) component 4624. The REC component 4624 is capable of outputting images to the DF 4602 , SAO 4604 , and ALF 4606 for filtering before storing the images in the reference picture buffer 4612 .

[0439] Fig.28 The method 2800 of an image or video implemented by a decoding device (eg, a decoder) according to an embodiment of the present disclosure is shown in FIG. 2802. In block 2802, the decoding device uses the bitstream and the output of the first sub-network to obtain a quantized residual potential value In an embodiment, the first sub-network is not autoregressive.

[0440] At block 2804, the decoding device uses the second sub-network and the quantized potential value The reconstructed sample points are used to obtain the predicted value mean. In block 2806, the decoding device uses the quantized residual sample points And the predicted mean mean to reconstruct the quantitative potential value The next sample point.

[0441] Fig.29 It is an image or video method 2900 implemented by an encoding device (eg, an encoder) according to an embodiment of the present disclosure. In block 2902, the encoding device first transforms an input image using an analysis transform to obtain a potential value y.

[0442] At block 2904, the encoding device uses the second sub-network and the quantized potential value The predicted value mean is obtained by using the reconstructed sample of w (when available). At block 2906, the encoding device subtracts the predicted value from the sample of the potential value to obtain the residual potential sample w.

[0443] At block 2908, the encoding device quantizes the residual potential samples and added to this predicted value to obtain a quantitative potential value In block 2910, the encoding device uses the first sub-network and the quantized residual potential sample In one embodiment, the first sub-network is not autoregressive.

[0444] A list of some example preferred solutions is provided below.

[0445] The following solutions illustrate examples of the techniques discussed herein.

[0446] 1. An image or video processing method implemented by a decoding device, comprising the steps of: using a bit stream and an output of a first sub-network to obtain a quantized residual potential value where the first subnetwork is not autoregressive; using the second subnetwork and the quantized potential value The reconstructed sample points are used to obtain the predicted value mean; and the quantized residual sample points are used and the predicted mean to reconstruct the quantized potential value The next sample point.

[0447] 2. The method according to claim 1 further comprises: using the quantized potential value and synthesize the transform to obtain the reconstructed image.

[0448] 3. An image or video method implemented by an encoding device, comprising the steps of: first transforming an input image using an analysis transform to obtain a potential value y; using a second sub-network and a quantized potential value The reconstructed sample points (if available) are used to obtain the predicted value mean; the predicted value is subtracted from the sample points of the potential value to obtain the residual potential sample point w; the residual potential sample point is quantized and added to the predicted value to obtain a quantitative potential value and using the first subnetwork and quantized residual potential samples The bit stream is obtained by sampling points, wherein the first sub-network is not autoregressive.

[0449] 4. The method according to any one of claims 1-3, wherein the first sub-network takes the first quantized super potential value as input and generates a probability parameter.

[0450] 5. The method according to any one of claims 1 to 4, wherein obtaining samples of quantized residual potential values ​​comprises entropy decoding, wherein the probability parameters and the bitstream are used as input.

[0451] 6. A method according to any one of claims 1-5, wherein obtaining the bitstream comprises entropy coding, wherein the probability parameter and the quantized residual potential value are input.

[0452] 7. The method according to any one of claims 1 to 6, wherein the probability parameter does not include a mean.

[0453] 8. The method according to any one of claims 1-7, wherein a zero mean probability distribution is used in entropy encoding or entropy decoding.

[0454] 9. The method according to any one of claims 1 to 8, wherein the second sub-network takes as input, in addition to the reconstructed samples of the quantized potential values, the second quantized super potential values.

[0455] 10. The method according to any one of claims 1-9, wherein the first quantified super potential value and the second quantified super potential value are the same.

[0456] 11. The method according to any one of claims 1-10, wherein the quantized super potential value is obtained from a bit stream in the decoder.

[0457] 12. The method according to any one of claims 1 to 11, wherein the quantized super potential value is obtained by using a sub-network from the potential value y or the quantized potential value Obtained in.

[0458] 13. The method according to any one of claims 1-12, wherein the second sub-network is autoregressive.

[0459] 14. The method according to any one of claims 1-13, wherein the second sub-network comprises a context module.

[0460] 15. The method according to any one of claims 1-14, wherein the second sub-network comprises a super decoder module.

[0461] 16. An apparatus for processing video data, comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform.

[0462] 17. A non-transitory computer-readable medium, comprising a computer program product for use by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, such that when the computer executable instructions are executed by a processor, the video codec device performs the method according to any one of claims 1-15.

[0463] 18. A non-transitory computer-readable recording medium storing a bit stream of a video generated by a method performed by a video processing device, wherein the method comprises the method according to any one of claims 1 to 15.

[0464] 19. A method for storing a bitstream of a video, comprising the method according to any one of claims 1-15.

[0465] 20. A method, apparatus or system as described in this document.

[0466] In the solution described herein, an encoder can comply with the format rules by generating a codec representation according to the format rules. In the solution described herein, a decoder can parse syntax elements in the codec representation using the format rules, knowing the presence and absence of syntax elements according to the format rules, to produce decoded video.

[0467] In this document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during the conversion from a pixel representation of a video to a corresponding bitstream representation, or vice versa. The bitstream representation of the current video block may correspond, for example, to bits that are co-located or distributed at different positions within the bitstream, as defined by the syntax. For example, macroblocks may be encoded based on transformed and coded error residual values ​​and may also use bits in the header and other fields in the bitstream. In addition, during the conversion, the decoder may parse the bitstream based on the determinations described in the above solution, knowing that certain fields may or may not be present. Similarly, the encoder may determine that certain syntax fields are included or not included, and generate the codec representation accordingly by including or excluding syntax fields from the codec representation.

[0468] The disclosed and other solutions, examples, embodiments, modules and functional operations described in this document may be implemented in digital electronic circuits, or in computer software, firmware or hardware, including the structures disclosed in this document and their structural equivalents, or a combination of one or more of them. The disclosed and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium, for execution by a data processing device or for controlling the operation of the data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a storage device, a material composition that affects a machine-readable propagation signal, or a combination of one or more of them. The term "data processing device" encompasses all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0469] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files storing one or more modules, subroutines, or code portions). A computer program may be deployed on a single computer for execution, or may be deployed on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network for execution.

[0470] The processes and logic flows described in this document may be performed by one or more programmable processors executing one or more computer programs to perform operations by operating on input data and generating output. The processes and logic flows may also be performed by, and devices may also be implemented as, special purpose logic circuitry (e.g., a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC)).

[0471] For example, processors suitable for executing computer programs include general and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, the computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operably coupled to receive data from one or more mass storage devices or to transfer data to them, or both. However, the computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and compact disk read-only memory (CDROM) and digital versatile disk read-only memory (DVD-ROM) disks. The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

[0472] Although this patent document contains many details, these details should not be interpreted as limitations on the scope of any subject matter or content that may be claimed, but rather as descriptions of features specific to a particular embodiment of a particular technology. Certain features described in the context of a separate embodiment in this patent document may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. In addition, although features may be described as working in certain combinations and even initially claimed in this manner above, one or more features from the claimed combination may be deleted from the claimed combination in some cases, and the claimed combination may point to a sub-combination or a change in a sub-combination.

[0473] Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in sequence, or that all of the operations shown be performed to achieve the desired results. In addition, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0474] Only a few implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and shown in this patent document.

[0475] A first component is directly coupled to a second component when there are no intermediate components other than a line, trace, or another medium between the first component and the second component. A first component is indirectly coupled to a second component when there are intermediate components other than a line, trace, or another medium between the first component and the second component. The term "coupled" and its variations include direct coupling and indirect coupling. Unless otherwise specified, the use of the term "about" is intended to include a range of ±10% of the number that follows it.

[0476] Although several embodiments have been provided in the present disclosure, it should be understood that the disclosed systems and methods may be implemented in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are considered to be illustrative rather than restrictive, and are not intended to be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0477] In addition, without departing from the scope of the present disclosure, the techniques, systems, subsystems and methods described and shown as discrete or separate in various embodiments may be combined or integrated with other systems, modules, techniques or methods. Other items shown or discussed as coupled may be directly connected, or may be indirectly coupled or communicated through certain interfaces, devices or intermediate components, whether electrically, mechanically or otherwise. Other examples of changes, substitutions and alterations may be determined by those skilled in the art, and these changes, substitutions and alterations may be made without departing from the spirit and scope disclosed herein.

[0478] The solutions listed in this disclosure may be used to compress an image, compress a video, compress a portion of an image, or compress a portion of a video.

[0479] Furthermore, the techniques, systems, subsystems, and methods described and illustrated in the various embodiments may be used to compress an image, compress a video, compress a portion of an image, or compress a portion of a video.

Claims

1. A method for processing an image or video, comprising the following steps: Use the bitstream and the output of the first subnetwork to obtain the quantized residual potential value , wherein the first sub-network is not autoregressive; Using the second subnetwork and quantizing the potential values The reconstructed sample points are used to obtain the predicted value mean; and Using the quantized residual samples and the predicted mean to reconstruct the quantized potential value The next sample point.

2. An image or video processing method, comprising the following steps: First, the input image is transformed using the analysis transformation to obtain the potential value y; Using the second subnetwork and quantizing the potential values The reconstructed sample points (when available) are used to obtain the predicted value mean; Subtracting the predicted value from the sample of the potential value to obtain a residual potential sample w; Quantify the residual potential samples and added to the predicted value to obtain a quantitative potential value ; and Use the first subnetwork and quantized residual potential samples The bit stream is obtained by sampling points, wherein the first sub-network is not autoregressive.

3. The method according to any one of claims 1 to 2, wherein: The quantified potential sample points Application context model.

4. The method according to any one of claims 1 to 3, wherein: The output of the context model is the input of the prediction fusion module.

5. The method according to any one of claims 1 to 4, wherein: The context submodule includes: a convolutional layer with a convolution kernel size of 4×3, the number of input channels is 192, the number of output channels is 384, and The input of the context submodule is And the output is fed to the prediction fusion submodule.

6. The method according to any one of claims 1 to 5, wherein: The output of the context submodule and the prediction fusion submodule is the mean μ, which includes forecast or estimate.

7. The method according to any one of claims 1 to 6, wherein: The context submodule and the prediction fusion submodule are jointly used to generate the prediction μ.

8. The method according to any one of claims 1 to 7, wherein: The number of samples considered as input is proportional to the size of the context submodule convolution kernel and the number of input channels of the context submodule convolution kernel.

9. The method according to any one of claims 1 to 8, wherein: The number of convolution kernel sizes is 4×3, which is 4 samples in the horizontal direction by 3 samples in the vertical direction, and the number of input channels is 192.

10. The method according to any one of claims 1 to 9, wherein: Increasing the number of input channels or the size of the convolution kernel will increase the The convolution kernel size includes the product of the size in the horizontal direction and the size in the vertical direction.

11. The method according to any one of claims 1 to 10, wherein: Increasing the number of the output channels of the context submodule increases the information obtained from the input samples, thereby improving prediction performance.

12. The method according to any one of claims 1 to 11, wherein: The number of output channels of the convolution kernel of the context submodule is 384, and wherein increasing the number of output channels results in the prediction μ being closer to 13. The method according to any one of claims 1 to 12, wherein: The convolution kernel size is denoted by K×L and the input / output channels are denoted by A and B respectively, and wherein increasing any one of A, B, K or L will cause the complexity of the context module to increase proportionally.

14. The method according to any one of claims 1 to 13, wherein: The context submodule is divided into two parallel convolution kernels.

15. The method according to any one of claims 1 to 14, wherein: A convolution kernel size of the first convolution layer is 3×2, and a size of the second convolution layer is 4×3, and wherein for both the first convolution layer and the second convolution layer, the number of input channels and the number of output channels are 192.

16. The method according to any one of claims 1 to 15, wherein: The outputs of the two convolutions are connected by a concat block to generate the output of layer 384.

17. The method according to any one of claims 1 to 16, wherein: The concat block is configured to stack two input channels on top of each other, and wherein the number of channels at the output of the concat block is the sum of the number of channels in the input of the concat block.

18. The method according to any one of claims 1 to 17, wherein: The prediction network includes the context submodule and the prediction fusion network.

19. The method according to any one of claims 1 to 18, wherein: The input potential samples are processed by at least two convolutional layers, the convolution kernel sizes of the two convolutional layers are O×P and M×N respectively, and wherein the convolution kernel sizes are different in at least one direction such that O is not equal to M or P is not equal to N.

20. The method according to any one of claims 1 to 19, wherein: The outputs of the two convolutional layers are concatenated and input to a prediction fusion network, and wherein the prediction fusion network outputs a predicted value μ as an estimate of a potential sample point.

21. The method according to any one of claims 1 to 20, wherein: The input potential samples are processed by a first convolutional layer with a convolution kernel size of M×N, and the output of the first convolutional layer is processed by at least a second convolutional layer with a convolution kernel size of K×L.

22. The method according to any one of claims 1 to 21, wherein: The output of the first convolutional layer and the output of the second convolutional layer are connected, wherein the outputs are fed to the prediction fusion module, and wherein the output of the prediction fusion submodule is μ, which is an estimate of the potential sample point.

23. The method according to any one of claims 1 to 22, wherein: The number of samples considered by the second convolutional layer is greater than the number of samples considered by the first convolutional layer, and the number of samples considered by the convolutional layer is called the receptive field.

24. The method according to any one of claims 1 to 23, wherein: The complexity of the context model is proportional to (K×L+M×N), rather than proportional to (K+M)×(L+N).

25. The method according to any one of claims 1 to 24, wherein: The potential sample point is fed to the first convolutional layer, where the residual latent samples is fed to a second convolutional layer, wherein the outputs of the first convolutional layer and the second convolutional layer are connected using the concat block and fed to the prediction fusion module, and wherein the output of the prediction fusion submodule is the μ as an estimate of the potential sample point.

26. The method according to any one of claims 1 to 25, wherein: The potential sample and the residual potential sample are both related to the potential sample to be predicted, and wherein the potential sample and the residual potential sample are both used as inputs of a predictor.

27. The method according to any one of claims 1 to 26, wherein: The output of the prediction fusion module is the potential sample point to be predicted. instead of the μ as an estimate of the potential sample point, and The potential sample point to be predicted is obtained by combining the predicted μ with the residual sample point obtained by adding or directly serving as the output of the prediction fusion module.

28. The method according to any one of claims 1 to 27, wherein: is a matrix or tensor comprising a plurality of potential samples, and wherein the input comprises samples that have been obtained by prediction.

29. The method according to any one of claims 1 to 28, wherein: The prediction module includes the context module and the prediction fusion module, and wherein the prediction module is an autoregressive neural network that sequentially processes the potential values ​​in order Sample point.

30. The method according to any one of claims 1 to 29, wherein: Fusion methods include addition.

31. The method according to any one of claims 1 to 30, wherein: Arithmetic encoding and decoding are performed independently of the first sub-network, and wherein the first sub-network is an autoregressive sub-network.

32. The method according to any one of claims 1 to 31, wherein: The fully sequential arithmetic encoding and / or decoding process is performed by the central processing unit.

33. The method according to any one of claims 1 to 32, wherein: The second sub-network is configured to estimate probability parameters used by an arithmetic encoding and / or decoding process.

34. The method according to any one of claims 1 to 33, wherein: The arithmetic coding and / or decoding process is used to encode and / or decode the quantized latent residual instead of the quantized latent value.

35. The method according to any one of claims 1 to 34, wherein: The autoregressive subnetwork is used only to estimate the mean of the latent value instead of estimating the mean and variance of the Gaussian distribution, and wherein the arithmetic encoder and decoder then only use the mean of the latent value to encode and decode the samples of the quantized latent value.

36. The method according to any one of claims 1 to 35, wherein: The entropy decoding and / or arithmetic decoding processes are performed independently, and wherein the decoding processes are performed by a central processing unit.

37. The method according to any one of claims 1 to 36, wherein: After entropy decoding is completed, the quantized residual potential values ​​are transmitted to the graphics processing unit.

38. The method according to any one of claims 1 to 37, wherein: The context and entropy parameter module is performed independently of arithmetic encoding and decoding, and wherein the context and entropy parameter module uses a graphics processing unit.

39. The method according to any one of claims 1 to 38, wherein: Idle processing time is eliminated.

40. The method according to any one of claims 1 to 39, wherein: The central processing unit and the graphics processing unit work together.

41. The method according to any one of claims 1 to 40, wherein: The decoding process is performed as follows: Performing an entropy decoding process to obtain all samples of the quantized potential residual; transmitting the obtained data to a graphics processing unit, wherein the transmitting is performed only once; executing a context and entropy parameter module by the graphics processing unit, wherein no idle waiting occurs in the graphics processing unit, and wherein the quantized potential value is obtained; as well as The synthetic transform is performed at the decoder to obtain the reconstructed image.

42. The method according to any one of claims 1 to 41, wherein: One or more sub-networks are utilized as super-encoders or super-decoders for super-information.

43. The method according to any one of claims 1 to 42, wherein: At least one sub-network is utilized to generate super-information, and wherein the super-information is dependent on the parsing process for the latent information.

44. The method according to any one of claims 1 to 43, wherein: At least one sub-network is utilized to generate super-information, and wherein the super-information is independent of the parsing process for the latent information.

45. The method according to any one of claims 1 to 44, wherein: At least one sub-network is utilized to generate super-information, and wherein the super-information is used to predict the underlying signal.

46. ​​The method according to any one of claims 1 to 45, wherein: The super information includes statistical information or probability distribution information of the potential signal, and wherein the potential signal is quantized.

47. The method according to any one of claims 1 to 46, wherein: The statistical information or probability distribution information includes a mean value of the underlying signal.

48. The method according to any one of claims 1 to 46, wherein: The statistical information or probability distribution information includes the variance of the underlying signal.

49. The method according to any one of claims 1 to 48, wherein: The latent signal can be encoded and decoded in a predictive manner.

50. The method according to any one of claims 1 to 49, wherein: In the encoder, y'=y-p is encoded, where y is the potential sample and p is the prediction.

51. The method according to any one of claims 1 to 50, wherein: In the decoder y*=y'+p is reconstructed.

52. The method according to any one of claims 1 to 51, wherein: y is quantized prior to the prediction process.

53. The method according to any one of claims 1 to 51, wherein: y is not quantized prior to the prediction process.

54. The method according to any one of claims 1 to 53, wherein: Prior to the prediction process, p is quantized.

55. The method according to any one of claims 1 to 53, wherein: p is not quantized prior to the prediction process.

56. The method according to any one of claims 1 to 55, wherein: After the prediction process, y' is quantized.

57. The method according to any one of claims 1 to 55, wherein: No quantization of y' is performed after the prediction process.

58. The method according to any one of claims 1 to 57, wherein: At least one sub-network can be utilized to generate the prediction p.

59. The method according to any one of claims 1 to 58, wherein: The prediction p for the current y or y* can be generated using at least one previously decoded y* or y'.

60. The method according to any one of claims 1-59, further comprising: The potential value is quantified using and synthesizing the transform to obtain the reconstructed image.

61. The method according to any one of claims 1 to 60, wherein: The first sub-network and the second sub-network are used as super decoders for super information.

62. The method according to any one of claims 1 to 61, wherein: At least one of the first sub-network and the second sub-network is used to generate super information, and wherein the super information relies on a parsing process for latent information.

63. The method according to any one of claims 1 to 61, wherein: At least one of the first sub-network and the second sub-network is used to generate super information, and wherein the super information is independent of a parsing process for underlying information.

64. The method according to any one of claims 1 to 63, wherein: At least one of the first sub-network and the second sub-network is used to generate super-information, and wherein the super-information is used to predict a potential signal.

65. The method of claim 64, wherein: The super information includes statistical information or probability distribution information of the potential signal, and wherein the potential signal is quantized.

66. The method of claim 65, wherein: The statistical information or probability distribution information includes a mean value of the underlying signal.

67. The method of claim 65, wherein: The statistical information or probability distribution information includes the variance of the underlying signal.

68. The method according to any one of claims 1 to 67, wherein: The potential signal is encoded or decoded in a predictive manner, and wherein the potential sample points encoded or decoded in the predictive manner are denoted as y'.

69. The method of claim 68, wherein: The potential signal is encoded and decoded according to y'=yp, where y is the potential sample and p is the prediction.

70. The method of claim 68, wherein: The latent signal is reconstructed according to y*=y'+p, where y* represents the reconstructed latent signal.

71. The method according to any one of claims 1 to 70, wherein: The potential samples are quantized prior to the prediction.

72. The method according to any one of claims 1 to 71, wherein: The potential samples are not quantized prior to the prediction.

73. The method according to any one of claims 1 to 72, wherein: The prediction is performed before quantization.

74. The method according to any one of claims 1 to 73, wherein: The prediction is not performed before quantization.

75. The method according to any one of claims 1 to 74, wherein: The potential samples encoded and decoded in the predictive manner are quantized after the prediction.

76. The method of any one of claims 1 to 75, wherein: The potential samples encoded and decoded in the predictive manner are not quantized after the prediction.

77. The method of any one of claims 1 to 76, wherein: The prediction is generated using at least one subnetwork including a first subnetwork and a second subnetwork.

78. The method of any one of claims 1 to 77, wherein: At least one of the reconstructed potential signal or the predictively encoded potential samples is used to generate a prediction for a current reconstructed potential signal or to generate a prediction for a current potential sample that is predictively encoded.

79. The method of claim 78, wherein: The first sub-network takes as input a first quantized super potential value and generates a probability parameter.

80. The method according to any one of claims 1 to 79, wherein The acquisition of samples of quantized residual potential values ​​comprises entropy decoding, wherein the probability parameters and the bitstream are used as input.

81. The method according to any one of claims 1 to 80, wherein: The obtaining of the bitstream comprises entropy coding, wherein the probability parameters and the quantized residual potential values ​​are taken as input.

82. The method of any one of claims 1 to 81, wherein: The probability parameters do not include the mean.

83. The method according to any one of claims 1 to 82, wherein: A zero-mean probability distribution is used in entropy encoding or decoding.

84. The method according to any one of claims 1 to 83, wherein: The second sub-network takes as input the second quantized super-potential value in addition to the reconstructed samples of the quantized potential value.

85. The method according to any one of claims 1 to 84, wherein The first quantized super potential value and the second quantized super potential value are the same.

86. The method of any one of claims 1 to 85, wherein: The quantized super potential value is obtained from a bitstream in the decoder.

87. The method of any one of claims 1 to 86, wherein: Use a subnetwork to quantize the potential value y from the potential value The quantitative super potential value is obtained in.

88. The method of any one of claims 1 to 87, wherein: The second sub-network is autoregressive.

89. The method according to any one of claims 1 to 88, wherein The second sub-network includes a context module.

90. The method according to any one of claims 1 to 89, wherein: The second sub-network includes a super decoder module.

91. An apparatus for processing video data, comprising: processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1-89.

92. A non-transitory computer-readable medium, comprising a computer program product for use by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, such that when the computer executable instructions are executed by a processor, the video codec device performs a method according to any one of claims 1-89.

93. A non-transitory computer-readable recording medium storing a bitstream of a video, the bitstream being generated by a method performed by a video processing device, wherein the method comprises the method according to any one of claims 1-89.

94. A method of storing a bitstream of a video, comprising the method according to any one of claims 1-89.

95. A method, apparatus or system as described in this document.

Images