Adaptive image and video compression method with variable rate and based on neural network

Abstract

A mechanism for processing video data in a neural network is disclosed. The mechanism includes obtaining quantized residual potential sample points. And processing the quantized residual potential sample points to obtain processed quantized residual potential sample points. A reconstructed potential sample point is then obtained based on the processed quantized residual potential sample point.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 390,614, filed on July 19, 2022, the teachings and disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] This patent document relates to the generation, storage and consumption of digital audio-visual media information in file format. Background Art

[0004] Digital video accounts for the largest share of bandwidth used on the Internet and other digital communications networks. As the number of networked user devices capable of receiving and displaying video increases, the demand for bandwidth used by digital video is likely to continue to grow. Summary of the invention

[0005] A first aspect relates to a method for processing video data in a neural network, the method comprising: obtaining quantized residual potential samples; performing a first processing on the quantized residual potential samples; and obtaining reconstructed potential samples based on the processed quantized residual potential samples.

[0006] A second aspect relates to a method for processing video data in a neural network, the method comprising: obtaining residual potential samples; performing a second process on the residual potential samples; and quantizing the processed residual potential samples to obtain quantized residual potential samples.

[0007] A third aspect relates to an apparatus for processing video data, the apparatus comprising: a processor; and a non-transitory memory having instructions thereon, wherein when the instructions are executed by the processor, the processor is caused to perform any one of the aforementioned aspects.

[0008] A fourth aspect relates to a non-transitory computer-readable medium, comprising a computer program product for use by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, such that when the instructions are executed by a processor, the video codec device performs a method according to any one of the preceding aspects.

[0009] A fifth aspect relates to a non-transitory computer-readable recording medium storing a bit stream by the method of any one of the preceding aspects.

[0010] For the purpose of clarity, any of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create new embodiments within the scope of the present disclosure.

[0011]

[0011] These and other features will become more clearly understood from the following detailed description, taken in conjunction with the accompanying drawings and claims, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] For a more complete understanding of the present disclosure, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0013] Figure 1 An example of a transform coding scheme is shown.

[0014] Figure 2 An example of quantization potential values ​​when using a super encoder / decoder is shown.

[0015] Figure 3 An example of a network architecture for an autoencoder implementing a hyper-prior model is shown.

[0016] Figure 4 A combined model is shown that jointly optimizes an autoregressive component that, together with a hyper-prior and an underlying autoencoder, estimates the probability distribution of a latent value from its causal context (e.g., a context model).

[0017] Figure 5 An example of an encoding process using a super encoder and a super decoder is shown.

[0018] Figure 6 An example of a decoding process is shown.

[0019] Figure 7 Shows a problem in the decoder network.

[0020] Figure 8 An example of an entropy encoding and decoding subnetwork in an image decoding architecture is shown.

[0021] Fig. 9 The decoding process according to the present disclosure is shown.

[0022] Fig.10 The encoding process according to the present disclosure is shown.

[0023] Fig.11 The encoding process according to the present disclosure is shown.

[0024] Fig.12 The encoding process according to the present disclosure is shown.

[0025] Fig.13 An embodiment of a coding and decoding device is shown.

[0026] Fig.14 An embodiment of a coding and decoding device is shown.

[0027] Fig.15An embodiment of a coding and decoding device is shown.

[0028] Fig.16 is a block diagram illustrating an example video processing system.

[0029] Fig.17 is a block diagram of an example video processing device.

[0030] Fig.18 is a flow chart of an example method of video processing.

[0031] Fig.19 is a block diagram illustrating an example video codec system.

[0032] Fig. 20 is a block diagram illustrating an example encoder.

[0033] Fig.21 is a block diagram illustrating an example decoder.

[0034] Fig. 22 is a schematic diagram of an example encoder.

[0035] Fig.23 is a flow chart of an example method of video processing.

[0036] Fig.24 is a flow chart of an example method of video processing. DETAILED DESCRIPTION

[0037] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or yet to be developed. The present disclosure should not be limited in any way to the exemplary implementations, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but may be modified within the scope of the appended claims and their full scope of equivalents.

[0038] Section headings are used in this document for ease of understanding and do not limit the applicability of the techniques and embodiments disclosed in each section to only that section. In addition, the techniques described herein are applicable to other video codec protocols and designs.

[0039] 1. Overview

[0040] A neural network-based image and video compression method includes an autoregressive subnetwork and an entropy coding and decoding engine, wherein the entropy coding and decoding is performed independently of the autoregressive subnetwork.

[0041] 2. Background

[0042] The past decade has witnessed the rapid development of deep learning in various fields, especially in computer vision and image processing. Inspired by the great success of deep learning techniques in the field of computer vision, many researchers have shifted their attention from conventional image / video compression techniques to neural image / video compression techniques. Neural networks were originally invented through interdisciplinary research in neuroscience and mathematics. They have shown strong capabilities in the context of nonlinear transformation and classification. Neural network-based image / video compression techniques have made significant progress during the past five years. It is reported that the latest neural network-based image compression algorithms [1] achieve rate-distortion (RD) performance comparable to that of the Versatile Video Codec (VVC) [2], which is the latest video coding standard developed by the Joint Video Experts Group (JVET) with experts from MPEG and VCEG. As the performance of neural image compression continues to improve, neural network-based video compression has become an actively developed research area. However, neural network-based video codecs still remain in their early stages due to the inherent difficulty of the problem.

[0043] 2.1 Image / Video Compression

[0044] Image / video compression generally refers to computing techniques that compress images / videos into binary codes for storage and transmission. The binary codes may or may not support lossless reconstruction of the original image / video, named lossless compression and lossy compression. Since lossless reconstruction is not necessary in most scenarios, most work is devoted to lossy compression. Usually, the performance of image / video compression algorithms is evaluated from two aspects, namely compression ratio and reconstruction quality. The compression ratio is directly related to the number of binary codes. The smaller the number of binary codes, the better the compression effect; the reconstruction quality is measured by comparing the reconstructed image / video with the original image / video. The higher the similarity, the better the reconstruction quality.

[0045] Image / video compression techniques can be divided into two branches, classical video codec methods and neural network-based video compression methods. Classical video codec schemes adopt transform-based solutions, in which researchers model the dependencies in the quantization mechanism by hand-crafted entropy codes to exploit statistical dependencies in latent variables (e.g., discrete cosine transform (DCT) or wavelet coefficients). Neural network-based video compression is available in two styles, namely, neural network-based codec tools and end-to-end neural network-based video compression. The former is embedded into existing classical video codecs as a codec tool and is only used as part of the framework, while the latter is an independent framework developed based on neural networks and does not rely on classical video codecs.

[0046] Over the past three decades, a series of classic video codec standards have been developed to adapt to the increasing visual content. The International Standards Organization International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) has two expert groups, the Joint Photographic Experts Group (JPEG) and the Moving Picture Experts Group (MPEG), and the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) also has its own Video Codec Experts Group (VCEG), which is responsible for the standardization of image / video codec technology. The influential video codec standards released by these organizations include JPEG, JPEG 2000, H.262, H.264 / AVC, and H.265 / HEVC. After H.265 / HEVC, the Joint Video Experts Group (JVET) composed of MPEG and VCEG has been working on a new video codec standard, Versatile Video Codec (VVC). The first version of VVC was released in July 2020. It is reported that VVC reduces the bit rate by an average of 50% at the same visual quality compared to HEVC.

[0047] Neural network-based image / video compression is not a new technology, as there are many researchers working on neural network-based image encoding and decoding [3]. However, the network architecture is relatively shallow and the performance is not satisfactory. Benefiting from the support of large amounts of data and powerful computing resources, neural network-based methods are better utilized in various applications. Currently, neural network-based image / video compression has shown promising improvements, proving its feasibility. However, the technology is still far from mature and many challenges need to be addressed.

[0048] 2.2 Neural Networks

[0049] A neural network, also known as an artificial neural network (ANN), is a computational model used in machine learning techniques that is typically composed of multiple processing layers, and each layer is composed of multiple simple but nonlinear basic computational units. One benefit of such deep networks is believed to be the ability to process data with multiple levels of abstraction and transform the data into different kinds of representations. Note that these representations are not manually designed; instead, the deep network including the processing layers is learned from massive amounts of data using general machine learning procedures. Deep learning eliminates the need for manual representations and is therefore believed to be useful, particularly for processing native unstructured data such as acoustic and visual signals, which has been a long-standing problem in the field of artificial intelligence.

[0050] 2.3 Neural Networks for Image Compression

[0051] Existing neural network approaches for image compression can be divided into two categories, namely pixel probability modeling and autoencoders. The former belongs to predictive encoding and decoding strategies, while the latter is a transform-based solution. Sometimes, these two approaches are combined in the literature.

[0052] 2.3.1 Pixel Probabilistic Modeling

[0053] According to Shannon information theory [6], the optimal method for lossless codec can achieve the minimum codec ratio -log2p(x), where p(x) is the probability of symbol x. Many lossless codecs have been proposed in the literature, and among them, arithmetic coding is considered to be the best [7]. Given a probability distribution p(x), arithmetic coding ensures that the codec ratio is as close to its theoretical limit -log2p(x) as possible without considering round-off errors. Therefore, the remaining question is how to determine the probability, which is very challenging for natural images / videos due to the curse of dimensionality.

[0054] Following the predictive coding strategy, one way to model p(x) is to predict pixel-by-pixel probability in raster scan order based on previous observations, where x is the image.

[0055] p(x)=p(x1)p(x2|x1)…p(x i |x1,…,x i-1 )…p(x m×n |x1,…,x m×n-1 ) (1)

[0056] Where m and n are the height and width of the image, respectively. The previous observation is also called the context of the current pixel. When the image is large, it may be difficult to estimate the conditional probability, so a simplified approach is to limit the scope of its context.

[0057] p(x)=p(x1)p(x2|x1)…p(x i |x i-k ,…,x i-1 )…p(x m×n |x m×n-k ,…,x m×n-1 ) (2)

[0058] Where k is a predefined constant that controls the scope of the context.

[0059] It should be noted that the condition may also consider sample values ​​of other color components. For example, when encoding and decoding red, green, and blue (RGB) color components, the R sample depends on the previously encoded pixel (including R / G / B samples), the current G sample can be encoded and decoded based on the previously encoded pixel and the current R sample, and when encoding and decoding the current B sample, the previously encoded pixel and the current R sample and the current G sample may also be considered.

[0060] Neural networks were initially introduced for computer vision tasks and have been shown to be effective in regression and classification problems. Therefore, it has been proposed to use neural networks to estimate a given its context x1,x2,…,x i-1 The probability p(x i ). In [7], the pixel probability of a binary image, ie, x i ∈{-1,+1}. The neural autoregressive distribution estimator (NADE) is designed for pixel probability modeling, which is a feedforward network with a single hidden layer. Similar work is presented in [8], where the feedforward network also has connections that skip the hidden layer and the parameters are also shared. Both [7] and [8] conduct experiments on the binary Modified National Institute of Standards and Technology (MNIST) dataset. For example, see the MNIST dataset: http: / / yann.lecun.com / exdb / mnist / . In [9], NADE is extended to a real-valued model real valued neural autoregressive density estimator (RNADE), where the probability p(x i |x1,…,x i-1 ) is derived using a Gaussian mixture. Their feed-forward network also has a single hidden layer, but the hidden layer has rescaling to avoid saturation and uses rectified linear units (ReLU)

[10] instead of sigmoid. In

[11] , NADE and RNADE are improved by reorganizing the order of pixels and by using deeper neural networks.

[0061] Designing advanced neural networks plays an important role in improving pixel probability modeling. In

[10] , a multidimensional long short-term memory (LSTM) was proposed, which works with conditional Gaussian scaled mixtures for probabilistic modeling. LSTM is a special kind of recurrent neural networks (RNNs) and is shown to be good at modeling sequence data. The spatial variant of LSTM was later used for images in

[13] . Several different neural networks have been studied, including RNNs and convolutional neural networks (CNNs), namely PixelRNN and PixelCNN. In PixelRNN, two variants of LSTM are proposed, called row LSTM and diagonal BiLSTM, where the latter is specifically designed for images. PixelRNN synthesizes residual connections to help train deep neural networks with up to 12 layers. In PixelCNN, masked convolutions are used to adapt to the shape of the context. Compared with previous work, PixelRNN and PixelCNN are more specialized for natural images: they consider pixels as discrete values ​​(e.g., 0, 1, ..., 255) and predict a multinomial distribution of discrete values; they process color images in the RGB color space; they work well on the large-scale image dataset ImageNet. In

[14] , Gated PixelCNN was proposed to improve PixelCNN and achieve comparable performance to PixelRNN but with much lower complexity. In

[15] , PixelCNN++ was proposed on PixelCNN with the following improvements: discretized logistic mixture likelihoods were used instead of 256-way multinomial distributions; downsampling was used to capture structures at multiple resolutions; additional shortcut connections were introduced to speed up training; dropout was used for regularization; RGB was combined for one pixel. In

[16] , PixelSNAIL was proposed, where random convolutions were combined with self-attention.

[0062] Most of the methods in compression directly model the probability distribution in the pixel domain. Some researchers also try to model the probability distribution as a conditional probability distribution of an explicit representation or a latent representation. That is, we can estimate

[0063]

[0064] Where h is an additional condition, and p(x)=p(h)p(x|h) means that modeling is divided into unconditional modeling and conditional modeling. The additional condition can be image label information or high-level representation.

[0065] 2.3.2 Autoencoder

[0066] Autoencoders are derived from the well-known work proposed by Hinton and Salakhutdinov

[17] . The method is trained for dimensionality reduction and consists of two parts: encoding and decoding. The encoding part converts the high-dimensional input signal into a low-dimensional representation, which usually has a reduced spatial size but a larger number of channels. The decoding part attempts to recover the high-dimensional input from the low-dimensional representation. Autoencoders enable automatic learning of representations and do not require hand-crafted features, which is also considered one of the most important advantages of neural networks.

[0067] Figure 1 An example of a transform coding scheme is shown. By analyzing the network g a The original image x is transformed to obtain the latent representation y. The latent representation y is quantized and compressed into bits. The number of bits R is used to measure the codec rate. Quantized latent representation Then through the synthesis network g s Perform inverse transform to obtain the reconstructed image By using the function g p Transform x and Distortion is computed in perceptual space.

[0068] It is intuitive to apply autoencoder networks to lossy compression. We just need to encode the latent representation learned from the trained neural network. However, adapting autoencoders to image compression is non-trivial because the original autoencoder is not optimized for compression and is therefore not efficient by directly using the trained autoencoder. In addition, there are other major challenges. First, the low-dimensional representation should be quantized before being encoded, but quantization is not differentiable, which is required in backpropagation when training neural networks. Second, the goals in the compression scenario are different because both distortion and ratio need to be considered. Estimating the ratio is challenging. Third, practical image coding and decoding schemes need to support variable rates, scalability, encoding speed / decoding speed, and interoperability. To address these challenges, many researchers have actively contributed to this field.

[0069] Prototype autoencoders for image compression such as Figure 1 As shown, it can be regarded as a transform coding strategy. The original image x passes through the analysis network y = g a (x) is transformed, where y is the latent representation that will be quantized and encoded. The synthesis network will transform the quantized latent representation Perform inverse transform to obtain the reconstructed image The framework is trained using the rate-distortion loss function, which is where D is x and The distortion between them, R is expressed from the quantization The rate calculated or estimated, and λ is the Lagrange multiplier. It should be noted that D can be calculated in the pixel domain or the perceptual domain. All existing research works follow this prototype, and the difference may only be the network structure or the loss function.

[0070] In terms of network structure, recurrent neural networks (RNN) and convolutional neural networks (CNN) are the most widely used architectures. In the RNN-related category, Toderci et al.

[18] proposed a general framework for variable rate image compression using recurrent neural networks (RNNs). They used binary quantization to generate codes and did not consider the rate during training. The framework actually provides scalable encoding and decoding capabilities, where RNNs with convolutional and deconvolutional layers were reported to perform well. Toderci et al.

[19] then proposed an improved version, namely an encoder using a neural network similar to pixel RNN (PixelRNN) to compress binary codes. It is reported that its performance is better than JPEG on the Kodak image dataset when using the multi-scale structural similarity (MS-SSIM) evaluation metric. Johnston et al.

[20] further improved the RNN-based solution by introducing a hidden state activation mechanism. In addition, an SSIM weighted loss function was designed and a spatially adaptive bitrate mechanism was enabled. When using MS-SSIM as the evaluation metric, they achieved better results than the Better Portable Graphics (BPG) on the Kodak image dataset. Coveell et al.

[21] supported spatially adaptive bitrate by training a blocking-tolerant RNN.

[0071] Ballé et al.

[22] proposed a general framework for rate-distortion optimized image compression. Multi-base quantization is used to generate integer codes and the rate is considered during training, i.e., the loss is a joint rate-distortion cost, which can be the mean square error (MSE) or other. They add random uniform noise during training to motivate quantization and use the differential entropy of the noise code as a proxy for the rate. They use generalized divisive normalization (GDN) as the network structure, which includes a linear mapping followed by a nonlinear parameter normalization. The effectiveness of GDN for image coding and decoding is verified in

[23] . Ballié et al.

[24] then proposed an improved version, in which they used 3 convolutional layers as the forward transform, each followed by a downsampling layer and a GDN layer. Therefore, they used 3 inverse GDN layers to motivate the simulation of the inverse transform, each followed by an upsampling layer and a convolution layer. In addition, an arithmetic coding method is designed to compress integer codes. It is reported that the performance is better than JPEG and JPEG 2000 on the Kodak dataset in terms of MSE. In addition, Balfé et al.

[25] improved the method by designing a variance hyper-prior in the autoencoder. They used a subnetwork h a Transform the potential representation y to z = h a (y) and z will be quantized and transmitted as side information. Therefore, using subnetwork h s From the quantitative auxiliary information Decoding to quantization The standard deviation of is further used during arithmetic coding and decoding of . On the Kodak image set, their method is slightly worse than BPG in terms of PSNR. D.Minnen et al.

[26] further exploit the structure in the residual space by introducing an autoregressive model to estimate both the standard deviation and the mean. In more recent work

[27] , Z.Cheng et al. used a Gaussian mixture model to further remove redundancy in the residual. The reported performance is comparable to VVC

[28] on the Kodak image set using PSNR as the evaluation metric.

[0072] 2.3.3 Super Prior Model

[0073] In the transform codec approach to image compression, the encoder subnetwork (Section 2.3.2) uses a parametric analysis transform The image vector x is transformed into a latent representation y, which is then quantized to form because Since it is a discrete value, entropy coding and decoding techniques such as arithmetic coding and decoding can be used to losslessly compress the discrete value and transmit it as a bit sequence.

[0074] Figure 2 shows an example of the quantization potential when using a super encoder / decoder. Figure 2 It can be clearly seen from the center-left image and the center-right image that There is significant spatial correlation between the elements of . It is noteworthy that their scales (center-right image) appear to be spatially coupled. In

[25] , another set of random variables is introduced to capture spatial dependencies and further reduce redundancy. In this case, the image compression network is Figure 3 Depicted in.

[0075] Figure 3 An example of a network architecture for an autoencoder that implements a hyper-prior model is shown. Figure 3 In the figure, the left side of the model is the encoder g a and decoder g s (Explained in Section 2.3.2). The right side is used to obtain The additional super encoder h a and super decoder h s In this architecture, the encoder inputs the input image x to g a , producing a response y with a spatially varying standard deviation. The response y is fed to h a , summarizing the standard deviation distribution in z. z is then quantized compressed and transmitted as auxiliary information. The encoder then uses the quantization vector to estimate σ, the spatial distribution of the standard deviation, and use it to compress and transmit the quantized image representation The decoder first recovers the compressed signal Then it uses h s to obtain σ, which provides the correct probability estimate for the successful recovery Then it feeds to g s to obtain the reconstructed image.

[0076] When super encoders and super decoders are added to image compression networks, the quantization latent value is reduced. of space redundancy. Figure 2 The rightmost image in the figure corresponds to the quantized potential values ​​when the super encoder / decoder is used. Compared with the center right image, the spatial redundancy is significantly reduced because the sample-point correlation of the quantized potential values ​​is less.

[0077] exist Figure 2, an image from the Kodak dataset is shown on the left; a visualization of the latent representation y for the image is shown on the center left; the standard deviation σ of the latent values ​​is shown on the center right; and the latent values ​​y after processing by the super-prior (super-encoder and decoder) networks are shown on the right.

[0078] Figure 3 The network architecture of the autoencoder implementing the super-prior model is shown. The image of the autoencoder network is shown on the left, and the right corresponds to the super-prior subnetwork. The analysis and synthesis transformations are denoted as g a and g a Q represents quantization (Q), and AE and AD represent arithmetic encoder (AE) and arithmetic decoder (AD), respectively. The super prior model consists of two sub-networks, the super encoder (denoted by h a denoted) and super decoder (with h s denoted by ). The hyper-prior model generates a quantitative hyper-potential value It includes information about quantifying potential values The probability distribution information of the sample points. is included in the bitstream and is related to are transmitted together to the receiver (decoder).

[0079] 2.3.4 Context Model

[0080] Although the hyper-prior model improves the quantitative potential value , but additional improvements can be obtained by using autoregressive models to predict quantitative potential values ​​from their causal context (Context Model).

[0081] The term "autoregressive" means that the output of a process is later used as input to that process. For example, the context model subnetwork generates one sample of potential values, which is later used as input to obtain the next sample.

[0082] The authors in

[26] use a joint architecture where both the super-prior model sub-network (super-encoder and super-decoder) and the context model sub-network are utilized. The super-prior and context models are combined to learn to quantize the latent value The probability model of is then used for entropy coding and decoding. Figure 4 As depicted in , the outputs of the context sub-network and the super decoder sub-network are combined by the sub-network and are called entropy parameters, which are parameters of the mean μ and variance σ of the Gaussian probability model. The Gaussian probability model is then used to encode the samples of the quantized potential values ​​into the bitstream with the help of the arithmetic encoder (AE) module. In the decoder, the Gaussian probability model is utilized to obtain the quantized potential values ​​from the bitstream through the arithmetic decoder (AD) module

[0083] Figure 4 A combined model is shown that jointly optimizes an autoregressive component that together with a hyper-prior and an underlying autoencoder estimates the probability distribution of the latent value from its causal context (context model). The real-valued latent representation is quantized (Q) to produce quantized latent values and quantify the super potential value The quantized latent values ​​and quantized super latent values ​​are compressed into a bitstream using an arithmetic encoder (AE) and decompressed by an arithmetic decoder (AD). The highlighted areas correspond to the components executed by the receiver (ie, decoder) to recover the image from the compressed bitstream.

[0084] Typically, the latent samples are modeled as Gaussian distributions or Gaussian mixture models (but not limited to). In

[26] , according to Figure 4 , the context model and the hyper-prior are jointly used to estimate the probability distribution of the underlying samples. Since a Gaussian distribution can be defined by a mean and a variance (also called sigma or scale), the joint model is used to estimate the mean and variance (denoted as μ and σ).

[0085] 2.3.5 Encoding process using joint autoregressive hyper-prior model

[0086] Figure 4 This corresponds to the compression method proposed in

[26] . In this section and the next section, the encoding process and the decoding process will be described separately.

[0087] Figure 5 The encoding process using a super encoder and a super decoder according to

[26] is shown. Figure 5 In Figure 1, the encoding process is depicted. The input image is first processed through the encoder subnetwork. The encoder transforms the input image into a transformed representation called a potential value (denoted by y). y is then input to the quantizer block denoted by Q to obtain a quantized potential value It is then converted into a bitstream (bits1) using an arithmetic coding block (denoted as AE). The arithmetic coding blocks are converted one by one in a sequential order. Each sample point of into the bit stream (bits1).

[0088] The modular super encoder, context, super decoder, and entropy parameter subnetworks are used to estimate the quantized latent value. The potential value y is input to the super encoder, which outputs the super potential value (denoted by z). The super potential value is then quantized The arithmetic encoding (AE) module is used to generate a second bit stream (bits2). The decomposition entropy module generates a probability distribution, which is used to encode the quantized super potential value into the bit stream. The quantized super potential value includes information about the quantized potential value. information about the probability distribution of .

[0089] The entropy parameter subnetwork generates probability distribution estimates, which are used to quantize the potential value The information generated by the entropy parameters usually includes the mean μ and variance σ parameters, which are used together to obtain a Gaussian probability distribution. The Gaussian distribution of a random variable x is defined as where the parameter μ is the mean or expectation of the distribution (as well as its median and mode), and the parameter σ is its standard deviation (or variance or scale). To define a Gaussian distribution, the mean and variance need to be determined. In

[26] , the entropy parameter module is used to estimate the mean and variance values.

[0090] The subnetwork superdecoder generates part of the information used by the entropy parameter subnetwork, and the other part of the information is generated by an autoregressive module called the context module. The context module generates information about the probability distribution of the samples of the quantized potential value using the samples that have been encoded by the arithmetic coding (AE) module. Quantized potential value It is usually a matrix consisting of many samples. The samples can be indicated using an index, such as depending on the matrix Dimensions or Sample Point The samples are encoded one by one by the AE, usually using a raster scan order. In raster scan order, the rows of the matrix are processed from top to bottom, where the samples in a row are processed from left to right. In this scenario (where the AE encodes the samples into the bitstream using a raster scan order), the context module uses the information generated by the context module and the super decoder to generate the information for converting the quantized potential values ​​into the bitstream. Probability distribution encoded into the bitstream (Bitstream 1).

[0091] Finally, as a result of the encoding process, the first bitstream and the second bitstream are transmitted to a decoder.

[0092] Note that other names can be used for the above modules.

[0093] In the above description, Figure 5 All elements in are collectively referred to as encoders. The analysis transformation that transforms the input image into a latent representation is also called an encoder (or autoencoder).

[0094] 2.3.6. Decoding process using the joint autoregressive hyper-prior model

[0095] Figure 6 A decoding process corresponding to

[26] is shown. Figure 6 The decoding process corresponding to

[26] is depicted respectively.

[0096] During the decoding process, the decoder first receives the first bit stream (bits1) and the second bit stream (bits2) generated by the corresponding encoder. Bits2 is first decoded by the arithmetic decoding (AD) module using the probability distribution generated by the factorization entropy subnetwork. The factorization entropy module usually generates the probability distribution using a predetermined template, such as a predetermined mean and variance value in the case of a Gaussian distribution. The output of the arithmetic decoding process of bits2 is It is the quantized super potential value. The AD process is restored to the AE process applied in the encoder. The AE and AD processes are lossless, which means that the quantized super potential value generated by the encoder is Can be reconstructed at the decoder without any changes.

[0097] In obtaining After that, it is processed by the super decoder, and the output of the super decoder is fed to the entropy parameter module. The three sub-networks, context, super decoder, and entropy parameter, employed in the decoder are the same as those in the encoder. Therefore, the exact same probability distribution is available in the decoder as in the encoder, which is essential for reconstructing the quantized latent values ​​without any loss. Therefore, the quantized potential value obtained in the decoder is The same version of can be obtained in the encoder.

[0098] The arithmetic decoding module decodes the samples of the quantized potential values ​​one by one from the bitstream bits1 after the probability distribution (e.g., mean and variance parameters) obtained by the entropy parameter subnetwork. From a practical point of view, the autoregressive model (context model) is inherently serial and therefore cannot be accelerated using techniques such as parallelization.

[0099] Finally, the fully reconstructed quantized potential value is input to the composite transform (in Figure 6 The decoder module is used to obtain the reconstructed image.

[0100] In the above description, Figure 6 All elements in are collectively referred to as the decoder. The synthetic transform that transforms the quantized latent values ​​into the reconstructed image is also called a decoder (or autodecoder).

[0101] 2.4 Neural Networks for Video Compression

[0102] Similar to conventional video codec technology, in neural network-based video compression, neural images serve as the basis for intra-frame compression, so the development of neural network-based video compression technology is later than that of neural network-based image compression technology, but more efforts are needed to solve the challenges due to its complexity. Since 2017, some researchers have been working on neural network-based video compression schemes. Compared with image compression, video compression requires effective methods to remove inter-picture redundancy. Inter-picture prediction is then a key step in these works. Motion estimation and compensation are widely adopted, but have not been implemented by trained neural networks until recently.

[0103] Research on neural network-based video compression can be divided into two categories according to the target scenario: random access and low latency. In the random access case, decoding can be started from any point in the sequence, usually the entire sequence is divided into multiple separate segments, and each segment can be decoded independently. In the low latency case, it aims to reduce the decoding time, so that usually only the previous frame in time can be used as a reference frame to decode the subsequent frame.

[0104] 2.4.1 Low Latency

[0105]

[29] first proposed a video compression scheme with a trained neural network. They first divided the video sequence frames into blocks, and each block will choose one of the two available modes, intra codec or inter codec. When intra codec is selected, there is an associated autoencoder to compress the block. When inter codec is selected, motion estimation and compensation are performed by traditional methods, and the trained neural network will be used for residual compression. The output of the autoencoder is directly quantized and encoded by the Huffman method.

[0106] Chen et al.

[31] proposed another neural network-based video codec scheme using PixelMotionCNN. Frames are compressed in temporal order and each frame is divided into blocks that are compressed in raster scan order. Each frame is first extrapolated by the two previous reconstructed frames. When a block is to be compressed, the extrapolated frame is fed into PixelMotionCNN along with the context of the current block to derive a latent representation. The residual is then compressed using a variable rate image scheme

[34] . The performance of this scheme is comparable to that of H.264.

[0107] Lu et al.

[32] proposed a video compression framework based on a truly end-to-end neural network, in which all modules are implemented by neural networks. The scheme accepts the current frame and the previously reconstructed frame as input, and derives the optical flow as motion information through a pre-trained neural network. The motion information will be warped together with the reference frame, and then the motion compensated frame is generated by the neural network. The residual and motion information are compressed through two separate neural autoencoders. The entire framework is trained with a single rate-distortion loss function. It achieves better performance than H.264.

[0108] Rppel et al.

[33] proposed an advanced video compression scheme based on advanced neural networks. It inherits and extends the traditional video codec scheme of neural networks with the following main features: 1) only one autoencoder is used to compress motion information and residuals; 2) motion compensation with multiple frames and multiple optical flows; 3) online states are learned and propagated through subsequent frames over time. This scheme has better performance than the HEVC reference software in terms of multi-scale structural similarity (MS-SSIM).

[0109] J.Lin et al.

[36] proposed an extended video compression framework based on an end-to-end neural network of

[32] . In this solution, multiple frames are used as references. Therefore, it is possible to provide a more accurate prediction of the current frame by using multiple reference frames and associated motion information. In addition, motion field prediction is deployed to remove motion redundancy along the temporal channel. A post-processing network is also introduced in this work to remove reconstruction artifacts from the previous process. The performance is better than

[32] and H.265 in terms of both peak signal-to-noise ratio (PSNR) and MS-SSIM.

[0110] Erikur et al.

[37] proposed a framework based on

[32] by replacing the scale-space flow of the commonly used optical flow with a scale parameter. They reportedly achieved better performance than H.264.

[0111] Z.Hu et al.

[38] proposed a multi-resolution representation for optical flow based on

[32] . Specifically, the motion estimation network generates multiple optical flows with different resolutions and lets the network learn which one to choose under a loss function. The performance is slightly improved compared to

[32] and better than H.265.

[0112] 2.4.2 Random Access

[0113] Wu et al.

[30] proposed a neural network-based video compression scheme with frame interpolation. The key frame is first compressed by a neural image compressor, and the remaining frames are compressed in a hierarchical order. They perform motion compensation in the perceptual domain, i.e., feature maps are derived at multiple spatial scales of the original frame and the motion is used to warp the feature maps, which are then used in the image compressor. The method is reported to be comparable to H.264.

[0114] Djuloah et al.

[41] proposed a method for interpolation-based video compression, where the interpolation model combines motion information compression and image synthesis, and the same autoencoder is used for both the image and the residual.

[0115] Airdessein et al.

[35] proposed a neural network-based video compression method based on variational autoencoders and deterministic encoders. Specifically, the model includes an autoencoder and an autoregressive prior. Unlike previous methods, this method accepts a group of pictures (GOP) as input and incorporates a 3D autoregressive prior while encoding and decoding the latent representation by considering temporal correlation. It provides performance comparable to H.265.

[0116] 2.5 Introduction

[0117] Almost all natural images / videos are in digital format. Grayscale digital images can be obtained by Indicates that is the set of pixel values, m is the image height and n is the image width. For example, is a public setting, and in this case Therefore a pixel can be represented by an 8-bit integer. An uncompressed grayscale digital image has 8 bits per pixel (bpp), while compressed ones must have fewer bits.

[0118] Color images are usually represented in multiple channels to record color information. For example, in the RGB color space, an image can be represented by To represent, there are three separate channels storing red, green and blue information. Similar to an 8-bit grayscale image, an uncompressed 8-bit RGB image has 24bpp. Digital images / videos can be represented in different color spaces. Neural network-based video compression schemes are mainly developed in the RGB color space, while traditional codecs typically use the YUV color space to represent video sequences. In the YUV color space, the image is decomposed into three channels, namely Y, Cb and Cr, where Y is the brightness component and Cb / Cr is the chrominance component. Since the human visual system is less sensitive to the chrominance component, the benefit comes from the fact that Cb and Cr are usually downsampled to achieve pre-compression.

[0119] A color video sequence consists of multiple color images, called frames, to record scenes at different time stamps. For example, in the RGB color space, a color video can be represented by X = {x0, x1, …, x t ,…,x T-1} represents, where T is the video sequence The number of frames in. If m = 1080, n = 1920, And the video has 50 frames per second (fps), then the data rate of the uncompressed video is 1920×1080×8×3×50=2,488,320,000 bits per second (bps), about 2.32 Gbps, which requires a lot of storage, so it must be compressed before transmission over the Internet.

[0120] Typically, for natural images, lossless methods can achieve a compression ratio of about 1.5 to 3, which is obviously lower than the requirement. Therefore, lossy compression came into being to achieve further compression ratios, but at the cost of causing distortion. Distortion can be measured by calculating the average squared difference between the original image and the reconstructed image, namely the mean square error (MSE). For grayscale images, MSE can be calculated by the following equation.

[0121]

[0122] Therefore, the quality of the reconstructed image compared to the original image can be measured by the Peak Signal-to-Noise Ratio (PSNR):

[0123]

[0124] in for The maximum value in , for example, is 255 for an 8-bit grayscale image. There are other quality assessment metrics such as structural similarity (SSIM) and multi-scale SSIM (MS-SSIM) [4].

[0125] To compare different lossless compression schemes, one only needs to compare the compression ratio for a given resulting rate, and vice versa. However, to compare different lossy compression methods, both the rate and the reconstruction quality must be considered. For example, a commonly used approach is to compute the relative rates for several different quality levels and then average these computed rates; the average relative rate is called the Bjontgaard rate of increase (BD-rate) [5]. There are other important aspects to evaluate image / video codec schemes, including encoding / decoding complexity, scalability, robustness, etc.

[0126] 3. Questions for discussion

[0127] 3.1 Core Issues

[0128] Many image compression networks include autoregressive models (e.g., context models) to improve compression performance. However, the autoregressive models alternate with an entropy decoding process that is inherently serial, resulting in a decoding process that is inherently serial (not efficiently parallelizable) and very slow.

[0129] Figure 7 The problem in the decoder network is shown, with the decoder network highlighted in the dashed box. This problem relates to the entropy decoding part of the prior art image decoding architecture. Figure 7 A prior art decoder design is depicted. The module on the right hand side, encapsulated in the dashed rectangle, is responsible for the entropy decoding of the quantized latent values Due to their serial nature, this part is very slow in some architectures.

[0130] Figure 8 An example entropy encoding / decoding subnetwork in an image decoding architecture is shown. In the image decoding architecture according to

[26] , the process of reconstructing the quantized latent is performed as follows:

[0131] 1. Quantize the hyper-latent values Processed by the hyper-decoder to generate the first part of information. The first part of information is fed into the entropy parameter module.

[0132] 2. Recursively and serially perform the following operations to reconstruct the samples of the quantized latent values as follows:

[0133] a. The context module uses the samples to generate the second part of information, where

[0134] i. n < j, or

[0135] ii. If n equals j, then m < i.

[0136] (The sample located at [m,n] is the reconstructed sample).

[0137] a. Using the first part of information and the second part of information, the entropy parameter module generates μ[i,j] and σ[i,j], which are the mean and variance of the Gaussian probability distribution respectively.

[0138] b. The arithmetic decoder decodes the samples from the bitstream using the probability distribution whose mean and variance are μ[i,j] and σ[i,j] respectively.

[0139] After reconstructing the quantized latent values according to the above flowchart it is processed by the synthesis transform (decoder) to obtain the reconstructed picture. According to the notation used in

[27] , the synthesis transform is called decoding. The entire above process (including Reconstruction of the image and reconstruction of the image) is also called decoding or decoder.

[0140] In the above, the sample points of the quantized potential value are given by Note that a sample point is not necessarily a scalar value, it may be a vector and may contain multiple elements. In the rest of the application, the sample point can be represented by or In the latter, “:” is used to indicate the presence of a third dimension and to emphasize that a sample point has multiple elements.

[0141] Quantifying potential value After the samples of are reconstructed, a synthesis transform (ie, decoder) is performed to obtain the reconstructed image.

[0142] As is apparent from the above description, the arithmetic decoding operation and the context module operation form a The decoding of is a completely serial operation. This means The samples cannot be reconstructed in parallel, they need to be reconstructed one by one.

[0143] The arithmetic decoding process (not limited to arithmetic coding, including most other entropy coding methods, such as range coding) is a computationally simple but essentially sequential operation. The reason is that the bit stream consists of a series of bits, and the bits need to be decoded one by one. The process is suitable for execution by a fast processing unit, such as a CPU.

[0144] On the other hand, the context module and the entropy parameter module are computationally intensive and highly parallelizable operations. They are more suitable to be executed by a massively parallel processing unit, such as a graphics processing unit (GPU).

[0145] When the context module and the entropy parameter module are interleaved with arithmetic decoding, a problem occurs in the prior art image codec architecture. As described in the above flowchart, Decoding of a sample of necessitates applying a context module, an entropy parameter module, followed by an arithmetic decoding module. The context module and the entropy parameter module are deep neural networks, which means they include a large number of operations. Arithmetic decoding is a relatively simple operation, however, it is completely serial. Performing a completely serial arithmetic decoding operation interleaved with complex "context" and "entropy parameter" operations significantly slows down the decoding process.

[0146] As a first example, if the processes of the context, arithmetic decoding, and entropy parameter modules are executed in a GPU, the following occurs.

[0147] 1. First, arithmetic decoding is used to obtain samples of the quantized potential value. Since arithmetic decoding is completely serial, only a single core of the GPU is used. During this time, all other GPU cores (there can be thousands of cores in a GPU) will be idle waiting. In addition, each core of the GPU is slow because they are not designed to perform calculations quickly.

[0148] 2. Once the samples of quantized potential values ​​are obtained, the context module and entropy parameter module are executed using multiple cores of the GPU. This second step can be performed efficiently because the context and entropy parameter modules are suitable for large-scale parallelization and are therefore suitable for execution at the GPU.

[0149] 3. Go to step 1 until all samples of the quantized potential value are decoded.

[0150] It can be appreciated that when decoding is performed on the GPU, the source of the slowdown is step 1.

[0151] As a second example, if the processes of the context, arithmetic decoding, and entropy parameter modules are executed on a CPU, the following occurs.

[0152] 1. First, arithmetic decoding is used to obtain samples of the quantized potential values. Since arithmetic decoding is completely serial, the CPU is very suitable and thus obtains the samples very quickly.

[0153] 2. Once the samples of quantized potential values ​​are obtained, the context and entropy parameter modules are executed. However, the CPU is not suitable for performing a large number of operations, it only includes a few processing cores. Therefore, this step will be very slow.

[0154] 3. Go to step 1 until all samples of the potential value are decoded.

[0155] When the process is executed on a CPU, the slowdown is caused by step 2, which is not suitable for execution on a CPU.

[0156] Finally, as a third example, we can consider that part of the decoding is performed on the CPU (arithmetic decoding) and part of the decoding is performed on the GPU (context and entropy parameters). In this case, the following occurs.

[0157] 1. First, a sample of the potential value is obtained using arithmetic decoding. Since arithmetic decoding is completely serial, the CPU is well suited and thus obtains the sample very quickly. During this time, the GPU remains idle.

[0158] 2. The acquired data is transferred to the GPU.

[0159] 3. Once the samples of potential values ​​are obtained, the context and entropy parameter modules are executed using multiple cores of the GPU. This second step can be performed efficiently because the context and entropy parameter modules are suitable for large-scale parallelization and are therefore suitable for execution at the GPU. During this period, the CPU remains idle.

[0160] 4. The obtained data (mean and variance values) are sent to the CPU.

[0161] 5. Go to step 1 until all samples of the potential value are decoded.

[0162] As can be appreciated, using the CPU and GPU in tandem is also not feasible due to the idle time of the CPU and GPU and the time required to transfer data back and forth between the CPU and GPU. In fact, this is the slowest implementation of the 3 options.

[0163] In short, a problem arises when computationally complex but massively parallelizable processes (such as deep neural networks, e.g. context modules or entropy parameter modules) need to be executed in an interleaved manner with simple but completely serial entropy encoding and decoding processes (such as arithmetic encoding and decoding or range encoding and decoding). Prior art image encoding and decoding networks such as in

[26] or [1] suffer from this problem, and thus the decoding process is very slow in this architecture.

[0164] A direct solution to the above problem would be to remove the autoregressive "context" module, with the result that the loop comprising the arithmetic decoder, context and entropy parameter modules can be eliminated. However, this will result in a substantial degradation of compression efficiency. The present disclosure achieves the same results as the direct solution without sacrificing compression efficiency.

[0165] 4. The present disclosure

[0166] The detailed techniques herein should be considered as examples to explain the general concept. These techniques should not be interpreted in a narrow sense. In addition, these techniques can be combined in any way.

[0167] 4.1. Objectives of this disclosure

[0168] The goal of the present disclosure is to deinterleave the arithmetic decoding process and the computationally complex deep neural network operations. In other words, the goal of the present disclosure is to decouple the arithmetic decoding process from the neural network-based module. Therefore, the arithmetic decoding process can be completed independently without the need for input from the neural network-based process. As a result, the speed of decoding is significantly increased.

[0169] 4.2. Core of this disclosure

[0170] 4.2.1. Decoding process

[0171] Fig. 9The decoding process according to the present disclosure is shown.

[0172] According to the present disclosure, the decoding operation is performed as follows:

[0173] 1. First, the second sub-network is used to use the quantized super potential value ( Fig. 9 In ) to estimate the probability parameters.

[0174] 2. The probability parameters (e.g., variance) generated by the second network are used to generate quantized residual potential values ​​(in Fig.10 In the expression ). The arithmetic decoder decodes the received bit stream based on the probability parameters and generates

[0175] 3. Perform the following steps in a loop until you get All elements of.

[0176] a. The first subnetwork is used to use the already obtained The sample points are used to estimate the potential value of The mean value parameter of .

[0177] b. Quantify the potential value of the residual and the average value is used to obtain The next element of .

[0178] 4. After obtaining After all samples of , a synthesis transformation can be applied, such as Figure 7 The decoder module in to obtain the reconstructed image.

[0179] exist Fig. 9 An example implementation of the present disclosure is depicted in (Decoding Process). Fig. 9 In , the first subnetwork includes context, prediction, and optional super decoder modules. The second network includes a super prior variance decoder module. The quantized super potential is and Figure 7 Compared to the example implementation shown in , the arithmetic decoding process is removed from the loop including arithmetic decoding, context and entropy parameters. In contrast, according to the present disclosure, the arithmetic decoding process is performed without using any input from the context and entropy parameter module, so the arithmetic decoding process (which is deinterleaved) can be performed independently. According to the present disclosure, the arithmetic decoding module has two inputs, the bitstream and the probability parameter (e.g., variance), which is the output of the super-prior variance decoder. The super-prior variance decoder uses the quantized super potential value Generate probability parameters. The arithmetic decoding process generates quantized residual potential values

[0180] After obtaining the residual latent value, a recursive prediction operation is performed to obtain the latent value Potential Value The sample points are obtained as follows.

[0181] 1. The autoregressive context module is used to use the sample points The first input to the prediction module is generated where the (m, n) pair is the index of the sample point where the potential value has been obtained.

[0182] 2. Optionally, the second input to the prediction module is obtained by using the super decoder and quantized super potential value Obtained.

[0183] 3. Using the first and second inputs, the prediction module generates the mean mean[:,i,j].

[0184] 4. Mean value mean[:,i,j] and quantized residual potential value are added together to obtain the potential value

[0185] 5. Repeat steps 1-4 for the next sample point.

[0186] Fig.10 The encoding process according to the present disclosure is shown. Fig. 9 In comparison, Fig.10 In , the same quantized super potential value is used as the input to the super decoder and super prior variance decoder modules. The rest of the operations are the same as above.

[0187] Whether and / or how to apply at least one method disclosed in the document may be signaled, for example, in a bitstream from an encoder to a decoder.

[0188] Alternatively, whether and / or how to apply at least one method disclosed in the document may be determined by a decoder based on codec information such as dimension, color format, etc.

[0189] 4.2.2. Encoding process

[0190] Fig.11 The encoding process according to the present disclosure is shown. According to the present disclosure, the encoding operation is performed as follows:

[0191] First, an analytical transform, such as applying the encoder in the figure, is performed to obtain all samples of the potential value y.

[0192] 1. First, perform the following steps in a loop until the residual potential value is quantized All elements of are obtained.

[0193] a. The first sub-network is used to use the quantized potential values ​​that have been obtained to estimate the mean parameters of the potential value y.

[0194] The mean is subtracted from the residual w to obtain the residual, which is quantized to obtain the quantized residual potential value

[0195] c. Add to the average to get the quantitative potential

[0196] 2. Secondly, the second sub-network is used to use the quantized super potential value to estimate probability parameters (such as variance).

[0197] 3. The entropy encoder module encodes the quantized residual latent values ​​into the bitstream using probability parameters.

[0198] exist Fig.11 An exemplary implementation of the present disclosure is described in (Encoding Process). Fig.11 In the example, the first subnetwork includes context, prediction and optional super decoder modules. The second network includes a super prior variance decoder module. Figure 5 ), the arithmetic coding process is removed from the loop including arithmetic coding, context and entropy parameters. In contrast, according to the present disclosure, the arithmetic coding process is performed without using any input from the context and entropy parameter module, so the arithmetic coding process (which is deinterleaved) can be performed independently. According to the present disclosure, the arithmetic coding module has two inputs, the quantized residual potential and a probability parameter (e.g., variance), which is the output of the super-prior variance decoder. The arithmetic coding process uses a probability model with a mean of zero. The super-prior variance decoder uses the super-potential Generate probability parameters. The arithmetic coding process generates a bit stream that is transmitted to the decoder.

[0199] According to the recursive prediction operation, the quantized residual potential The sample points are as follows.

[0200] 1. The autoregressive context module is used to use the sample points The first input to the prediction module is generated, where the (m, n) pair is the index of the potential sample point that has been obtained.

[0201] 2. Optionally, the second input to the prediction module is obtained by using the super decoder and the super potential value Obtained.

[0202] 3. Using the first and second inputs, the prediction module generates the mean mean[:,i,j].

[0203] 4. Subtract the mean mean[:,i,j] from the latent value y[:,i,j] to obtain the residual latent value w[:,i,j].

[0204] 5. The residual potential value is quantized to obtain the quantized residual potential value

[0205] 6. is added to mean[:,i,j] to obtain the quantized potential value Repeat steps 1-5 for the next sample point.

[0206] Once the quantized residual potential value is obtained according to the above recursive process For all samples of The second sub-network (super-prior variance decoder) is used to estimate the probability parameters used in the entropy coding process.

[0207] Fig.12 The encoding process according to the present disclosure is shown. Fig.11 In comparison, Fig.12 In , the same quantized super potential value is used as the input to the super decoder and super prior variance decoder modules. The rest of the operations are the same as above.

[0208] 4.3 Some Differences Between the Prior Art and the Present Disclosure

[0209] There are at least three differences between the prior art and the present disclosure.

[0210] 1. Perform arithmetic coding and decoding independently of the autoregressive sub-network (first sub-network). In this way, the fully sequential arithmetic coding / decoding process can be performed by a fast processing unit, such as a CPU. A new sub-network (second sub-network) is introduced to estimate the probability parameters used by the arithmetic coding / decoding process.

[0211] 2. The arithmetic coding / decoding process is used to encode / decode the quantized potential value residual, rather than the quantized potential value as in the prior art.

[0212] 3. The autoregressive subnetwork is only used to estimate the mean of the potential value. In the prior art, the autoregressive subnetwork is used to estimate the mean and variance of the Gaussian distribution, which is then used by the arithmetic encoder and decoder to encode / decode samples of the quantized potential value.

[0213] Benefits of the present disclosure.

[0214] The benefits of the present disclosure are at least as follows.

[0215] 1. Entropy decoding (e.g., arithmetic decoding) process, which is a simple but completely serial operation that can be performed independently. For example, the entropy decoding process can be performed by a processing unit suitable for quickly performing serial operations, such as a CPU. Once the entropy decoding operation is completed, the obtained data (quantized residual potential) can be transferred to the GPU.

[0216] 2. Computationally heavy but easily parallelizable modules (such as context and entropy parameter modules) can be performed independently of arithmetic coding / decoding. For example, a processing unit suitable for massively parallel processing (such as a GPU) can be used to perform these operations.

[0217] 3. Eliminate idle processing time.

[0218] According to the present disclosure, the CPU and GPU can be used in series. For example, the decoding process can be performed as follows.

[0219] 1. First, perform and complete the entropy decoding process. Complete the entire process and obtain all the samples of the quantized potential residual.

[0220] 2. Transfer the acquired data to the GPU. Data transfer only happens once.

[0221] 3. The context and entropy parameter modules are executed and completed by the GPU. No idle waiting occurs in the GPU. The quantized potential value is obtained.

[0222] 4. Perform synthesis transformation (decoder) to obtain the reconstructed image.

[0223] Compared with the third example in Section 3.2, when both the CPU and GPU are used for decoding, the present disclosure eliminates the back-and-forth data transmission between the CPU and the GPU. In addition, it eliminates the idle waiting time.

[0224] In fact, the present disclosure can significantly increase the decoding speed by a factor of 10. Moreover, by means of a clever design, it does not suffer any loss in compression efficiency.

[0225] 4.6 Example Embodiments of the Present Disclosure

[0226] The detailed embodiments of the present disclosure described below should be considered as examples for explaining the general concept. These embodiments should not be interpreted in a narrow sense. In addition, these embodiments can be combined in any way.

[0227] 1. In one example, more than one sub-network is utilized as a super encoder / decoder for super information.

[0228] a. In one example, at least one sub-network is used to generate super information, which relies on a parsing process for latent information.

[0229] b. In one example, at least one sub-network is utilized to generate super-information that is independent of a parsing process for underlying information.

[0230] c. In one example, at least one sub-network is utilized to generate super information for predicting a potential signal.

[0231] d. In one example, the super information may include statistical information or probability distribution information of the underlying signal that may be quantified.

[0232] i. Statistical information or probability distribution information may include the mean of the underlying signal.

[0233] ii. Statistical information or probability distribution information may include the variance of the underlying signal.

[0234] 2. In one example, the latent signal may be encoded or decoded in a predictive manner.

[0235] a. In one example, y'=yp can be encoded at the encoder, where y is the potential sample and p is the prediction.

[0236] i. Correspondingly, y*=y'+p can be reconstructed at the decoder.

[0237] b. In one example, y may be quantized prior to the prediction process.

[0238] c. In one example, y may not be quantized before the prediction process.

[0239] d. In one example, p can be quantized before the prediction process.

[0240] e. In one example, p may not be quantized before the prediction process.

[0241] f. In one example, y' may be quantized after the prediction process.

[0242] g. In one example, y' may not be quantized after the prediction process.

[0243] h. In one example, at least one sub-network may be utilized to generate a prediction p.

[0244] i. In one example, at least one previously decoded y* or y' may be utilized to generate a prediction p of the current y or y*.

[0245] An embodiment having variable rate capability.

[0246] According to the present disclosure, in the encoder, the residual latent value w is processed before quantization. Fig.14 For example, the encoder according to the present disclosure is designed in Fig.14 According to the present disclosure, the residual potential value is first obtained by subtracting the predicted sample μ (also called the mean) from the potential sample y. Then, the residual potential sample is processed by module R, the input of module R is w, and the output of module R is ws. Instead of the residual sample, the processed residual sample ws is quantized to obtain the quantized residual sample The quantized residual samples are then included in the bitstream. By module T (in Fig.14 ) is further modified to obtain It is added to the predicted sample to obtain the potential sample

[0247] In the encoder, module R and module T are used in tandem, before and after quantization, to adjust the amplitude of the residual samples. Thus, the amount of information included in the bitstream can be achieved. For example, if module R reduces the amplitude of the residual samples, the bit rate required to encode the residual samples will be reduced. Therefore, the size of the bitstream required to encode the residual samples will be reduced, and thus the compression ratio will be increased. On the other hand, increasing the compression ratio will reduce the quality of the compressed image. Therefore, module R can be used to control the compression ratio, which is an important characteristic required by a compression system.

[0248] Fig.14 The module T in works in the opposite way to the module R. If the module R normally reduces the amplitude of the residual samples, the module T counteracts this by increasing the strength of the quantized residual samples. The amplitude can remain constant on average.

[0249] Fig.13 An embodiment of a coding and decoding device is shown. In the decoder ( Fig.13 ), the module T implemented in the encoder can also be implemented. Fig.13 An example implementation is depicted in . First, quantized residual samples are obtained from the bitstream after, Processed by module T to obtain the processed quantized residual samples The processed quantized residual samples are then added to the predicted samples to obtain the potential samples

[0250] The process performed by modules R and T may be a sample-by-sample multiplication with a vector. The sample-by-sample multiplication process may be performed as follows:

[0251] O[i]=T[i]×v[i]

[0252] According to the above formula, the i-th element of the output O is obtained by multiplying the i-th element of the vector T and the vector v. Similarly, in Fig.13 and Fig.14In the quantized residual sample after processing It can be obtained by the following formula:

[0253]

[0254] in is the quantized residual sample, and T is the processing A vector (or vectors) of .

[0255] exist Fig.14 In the encoder in, the processing module R can be applied in a similar way to obtain the processed residual sample ws, such as element-by-element multiplication:

[0256] ws[i]=R[i]×w[i]

[0257] ws The difference between them is that the latter are quantized samples (obtained after the quantization process).

[0258] The process T can also be performed as follows:

[0259]

[0260] The output is obtained by multiplying the i-th element of vector T1 with the i-th element of the quantized residual sample and adding the i-th sample of vector T2. The i-th element of .

[0261] According to an example, the sample points of the processed quantized residual sample points can be obtained according to the following formula:

[0262]

[0263] Where K is an index value that determines the index of the processing vector T[K] used to process the quantized residual samples. In other words, the value of K can be used to determine which vector (out of a set of possible vectors) is used to process the quantized residual samples.

[0264] According to the present disclosure, an indication may be included in the bitstream to indicate which vector from a set of vectors is used to process the quantized residual samples.

[0265] According to the present disclosure, the processed quantized residual sample points can be obtained according to the equation

[0266]

[0267] According to the above equation, by quantizing the residual sample point The processed quantized residual samples are obtained by multiplying the processed vector T[K], whose positions are given by the three indices i, x and y. According to an example embodiment, the processed quantized residual samples are obtained according to the first vector T[K]. At least one first sample point in , and obtain according to the second vector T[M] At least one second sample point in . The samples are then used to obtain potential samples The latent samples are then transformed to obtain a reconstructed image.

[0268] According to the present disclosure, the bitstream includes at least two indications to indicate the indexes of at least two processing vectors (or vectors) T1 and T2. T1 and T2 are used to process the quantized residual samples. To obtain It is later added to the predicted sample μ to obtain the potential sample The latent samples are then processed through a transform to obtain a reconstructed image.

[0269] According to one example, potential sample points is divided into at least 2 slices. A slice is a rectangular partition of potential samples. The first processing vector T1 is used to obtain The sample points are then used to obtain The samples corresponding to the first slice. And the second processing vector T2 is used to obtain The sample points are then used to obtain Additionally or alternatively, two indications may be included in the bitstream to indicate which processing vector corresponds to the first slice and which vector corresponds to the second slice.

[0270] Additionally or alternatively, samples of the probability parameter σ may be processed by a module P. Module P performs in a similar manner to modules T and R exemplified above (eg, sample-wise multiplication with a vector σ).

[0271] Fig.15 An embodiment of a coding and decoding device implemented according to the present disclosure is shown. Fig.15 In the example, the probability parameter σ is processed by the first module P. The processed probability parameter is used to obtain the quantized residual sample point Then it is processed by the second module T to obtain the processed quantized residual samples, which are then used to obtain Thus, a reconstructed image is obtained.

[0272] The reason for applying the P module to the probability parameters and the T module to the quantized residual samples is that the modification of the quantized residual samples results in different statistical properties. Therefore, it may be inefficient to estimate the statistical properties of the modified residual samples using the unmodified probability parameters. Therefore, according to the present disclosure, the probability parameters are also modified by the module P to align the modification of the quantized residual samples.

[0273] According to the present disclosure, an indication may be included in the bitstream to select a processing vector T (which is used to modify the quantized residual samples) and a processing vector P that modifies the probability parameter σ. In other words, a single index value may be used to determine which vector P and which vector T to use.

[0274] At the decoder side, the residual Scaling by parameter β produces The residual tensor is then scaled in the invRVS (Inverse Residual and Variance Scale) module (Section G.3) to form the residual tensor This is used to reconstruct the latent tensor

[0275] D. 4Sigma Scaling

[0276] This process applies on both the encoder and decoder side and is therefore normative.

[0277] Inputs to the process

[0278] - positive gain vector m (derived as described in F.2);

[0279] - The standard deviation tensor σ of size [C,h4,w4], which is the output of the super-prior variance decoder (D.3).

[0280] The output of this process

[0281] - Scaling by a positive gain vector standard deviation tensor σ′ of shape [C,h4,w4], which arrives at adaptive sigma scaling (G.3.3).

[0282] Sigma scaling modifies the standard deviation tensor σ[C,h4,w4] as input and multiplies the standard deviation tensor by the gain vector as output:

[0283] σ′[c,i,j]=m[c]·σ[c,i,j];i=0,…,h4-1,j=0,…,w4-1,c=0,…,C-1

[0284] All elements of a tensor in the same channel are scaled by the same multiplier.

[0285] F.3 Gain Unit

[0286] This process is applied only on the encoder side, so it is a non-normative process, described here to simplify the explanation of the normative inverse gain unit (F.4) and sigma scaling (F.5) processes.

[0287] Inputs to the process

[0288] - positive gain vector m (derived as described in F.2);

[0289] - A residual tensor r of shape [C, h4, w4], which is the result of subtracting the mean from the latent tensor, r = y - μ.

[0291] The output of this process

[0292] - scaled by a positive gain vector residual tensor r′ of shape [C,h4,w4]

[0293] On the encoder side, the gain unit is placed after the residual calculation ( Figure 4 .4-2). The output scaled residual tensor is equal to multiplying the input tensor by the gain vector in the channel dimension:

[0294] r′[c,i,j]=(m[c]·r[c,i,j])>>(gainVectorPrecision+extraMultiplierPrecision); 0≤i <h4,0≤j<w4,0≤c<C

[0295] All elements of a tensor in the same channel are scaled by the same multiplier.

[0296] F.4 Inverse Gain Unit

[0297] This process is applied at the decoder side and is therefore normative.

[0298] Inputs to the process

[0299] -Inverse gain vector m -1 (derived as described in F.2);

[0300] - reconstructed residual tensor of shape [C,h4,w4] It is the output of the decoder size SKIP process (G.4.3).

[0301] The output of this process

[0302] - The residual tensor consisting of the inverse gain vector of shape [C,h4,w4] Scaling is performed (which then enters the inverse residual scaling process G.3.4).

[0303] On the decoder side, the inverse gain unit is placed after the entropy decoder and takes the reconstructed residual as input ( Figure 4 .5-2). The output is the reconstructed residual tensor multiplied by the inverse gain vector:

[0304]

[0305] 0≤i <h4,0≤j<w4,0≤c<C

[0306] All elements of a tensor in the same channel are scaled by the same multiplier.

[0307] F.2 Control Parameters and Gain Vector Derivation

[0308] A total of four models are trained for different quality ranges. The model is selected based on the modelIdx encoded in the picture header. The rate control parameter β controls the compression ratio and defines the gain unit, inverse gain unit and Sigma scaling (in Figure 4 .4-2 and Figure 4 The operation marked as sGain in .5-2.

[0309] On the encoder side, the positive gain vector m is used in the gain unit, and in both the encoder and decoder, the positive gain vector m is used in the sigma scaling (sGain). On the decoder side, the inverse gain vector m -1 Used in the inverse gain unit. Positive gain vector m and inverse gain vector m -1 All have a dimension [C] equal to the number of channels of the residual tensor.

[0310] The learnable model includes the corresponding β∈{β t}'s reference positive gain vector and inverse gain vector Yes. Vector m t The elements of are 6-bit values.

[0311] The input to the gain vector derivation process is

[0312] -10 bit variable β displacement For the major component it is equal to beta_displacement_y, and for the minor component it is equal to beta_displacement_uv;

[0313] - Reference forward gain vector and inverse gain vector and the corresponding β∈{β t}.

[0314] The output of the gain vector derivation process is

[0315] - positive gain vector m;

[0316] -Inverse gain vector m -1 .

[0317] The gain vector is calculated by a simple scaling process:

[0318] m=m t β displacement ;m -1 =m -1 t 1 / β displacement .

[0319] In further calculations, extraMultiplierPrecision=5, gainVectorPrecision=5.

[0320]

[0321]

[0322] The following references are incorporated herein by reference as if reproduced in their entirety:

[0323] [1] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 7939–7948, 2020.

[0324] [2] B. Bross, J. Chen, S. Liu, and Y.-K. Wang, “Versatile Video Draft (Draft10),” JVET-S2001, July 2020.

[0325] [3] R. D. Dony and S. Haykin, “Neural network approaches to image compression,” IEEE Conference on Computer Vision, vol. 83, no. 2, pp. 288–303, 1995.

[0326] [4] Z. Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004.

[0327] [5] G.Bjontegaard's "Calculation of average PSNR differences between RD-curves", VCEG, Tech.Rep.VCEG-M33, 2001.

[0328] [6] C.E. Shannon, “Amathematical theory of communication,” Bell System Technical Journal, vol. 27, no. 3, pp. 379–423, 1948.

[0329] [7] H. Larochelle and I. Murray, “The neural autoregressive distribution estimator,” in Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, 2011, pp. 29–37.

[0330] [8] K. Gregor and Y. LeCun, “Learning representations by maximizing compression”, arXiv preprint arXiv:1108.1169, 2011.

[0331] [9] B. Uria, I. Murray, and H. Larochelle, “RNADE: The real-valued neural autoregressive density-estimator,” NIPS, 2013, pp. 2175–2183.

[0332]

[10] L. Theis and M. Bethge, “Generative image modeling using spatial LSTMs,” NIPS, 2015, pp. 1927–1935.

[0333]

[11] B. Uria, I. Murray, and H. Larochelle, “A deep and tractable density estimator,” in ICML, 2014, pp. 467–475.

[0334]

[12] IH Witten, RM Neal, and JG Creary, “Arithmetic coding for data compression”, Communications of the ACM, vol. 30, no. 6, pp. 520–541, 1987.

[0335]

[13] A. van den Oord, N. Kalchbrenner, and K. Kavukcuoglu, “Pixel recurrent neural networks,” in ICML, 2016, pp. 1747–1756.

[0336]

[14] "Conditional image generation with PixelCNN decoders" by A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, and K. Kavukcuoglu, NIPS, 2016, pp. 4790–4798.

[0337]

[15] T. Salimans, A. Karpathy, X. Chen, and D.P. Kingma, “PixelCNN++: Improving the PixelCNN with discretized logistic mixture likelihood and other modifications”, arXiv preprint arXiv:1701.05517, 2017.

[0338]

[16] X. Chen, N. Mishra, M. Rohaninejad, and P. Abbeel, “PixelSNAIL: An improved autoregressive generative model,” in ICML, 2018, pp. 863–871.

[0339]

[17] G.E. Hinton and R.R. Salakhutdinov, “Reducing the dimensionality of data with neural networks”, Science, vol. 313, issue 5786, pp. 504–507, 2006.

[0340]

[18] G. Toderici, S. M. O’Malley, S. J. Hwang, D. Vincent, D. Minnen, S. Baluja, M. Covell, and R. Sukthankar, “Variable rate image compression with recurrent neural networks”, arXiv preprint arXiv:1511.06085, 2015.

[0341]

[19] G. Toderici, D. Vincent, N. Johnston, S. J. Hwang, D. Minnen, J. Shor, and M. Covell, “Full resolution image compression with recurrent neural networks,” CVPR, 2017, pp. 5306–5314.

[0342]

[20] "Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks" by N. Johnston, D. Vincent, D. Minnen, M. Covell, S. Singh, T. Chinen, S. Jin Hwang, J. Shor and G. Toderici, CVPR, 2018, pp. 4385–4393.

[0343]

[21] M. Covell, N. Johnston, D. Minnen, S. J. Hwang, J. Shor, S. Singh, D. Vincent, and G. Toderici, “Target-quality image compression with recurrent,convolutional neural networks,” arXiv preprint arXiv:1705.06687, 2017.

[0344]

[22] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimization of nonlinear transform codes for perceptual quality,” PCS. IEEE, 2016, pp. 1–5.

[0345]

[23] J. Ballé, “Efficient nonlinear transforms for lossy image compression,” in PCS, 2018, pp. 248–252.

[0346]

[24] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimized image compression,” in International Conference on Learning Representations, 2017.

[0347]

[25] J. Ballé, D. Minnen, S. Singh, S. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” in International Conference on Learning Representations, 2018.

[0348]

[26] "Joint Autoregressive and Hierarchical Priors for Learned Image Compression" by D.Minnen, J.Ballé, and G.Toderici, arXiv.1809.02736.1, 2, 3, 4, 7.

[0349]

[27] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

[0350]

[28] Github repository “CompressAI: https: / / github.com / InterDigitalInc / CompressAI”, InterDigital Inc, accessed December 2020.

[0351]

[29] T. Chen, H. Liu, Q. Shen, T. Yue, X. Cao, and Z. Ma, “DeepCoder: A deep neural network based video compression,” in VCIP. IEEE, 2017, pp. 1–4.

[0352]

[30] C.-Y. Wu, N. Singhal, and P. Krahenbühl, “Video compression through image interpolation,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 416–431.

[0353]

[31] Z. Chen, T. He, X. Jin, and F. Wu, “Learning for video compression,” IEEE Transactions on Circuits and Systems for Video Technology, DOI: 10.1109 / TCSVT.2019.2892608, 2019.

[0354]

[32] G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai, and Z. Gao, “DVC: ​​An end-to-end deep video compression framework,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

[0355]

[33] O. Rippel, S. Nair, C. Lew, S. Branson, A. Anderson, and L. Bourdev, “Learned Video Compression,” in 2019 IEEE / CVF International Conference on Computer Vision (ICCV), Seoul, South Korea, 2019, pp. 3453–3462. doi: 10.1109 / ICCV.2019.00355.

[0356]

[34] G. Toderici, D. Vincent, N. Johnston, S. J. Hwang, D. Minnen, J. Shor, and M. Covell, “Full resolution image compression with recurrent neural networks,” CVPR, 2017, pp. 5306–5314.

[0357]

[35] A. Habibian, T. Rozendaal, J. Tomczak, and T. Cohen, “Video Compression with Rate-Distortion Autoencoders,” in Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), 2019, pp. 7033–7042.

[0358]

[36] J. Lin, D. Liu, H. Li, and F. Wu, “M-LVC: Multiple frames prediction for learned video compression,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

[0359]

[37] E. Agustsson, D. Minnen, N. Johnston, J. Ballé, S. J. Hwang, and G. Toderici, “Scale-Space Flow for End-to-End Optimized Video Compression,” in 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 8500–8509. doi: 10.1109 / CVPR42600.2020.00853.

[0360]

[38] X. Hu, Z. Chen, D. Xu, G. Lu, W. Ouyang, and S. Gu, “Improving deep video compression by resolution-adaptive flow coding,” in European Conference on Computer Vision (ECCV), 2020.

[0361]

[39] "λDomain Rate Control Algorithm for High" by B.Li, H.Li, L.Li and J.Zhang

[0362] Efficiency Video Coding,” IEEE Transactions on Image Processing, Vol. 23, No. 9, pp. 3841–

[0363] 3854 pages, September 2014, doi:10.1109 / TIP.2014.2336550.

[0364]

[40] L. Li, B. Li, H. Li and C. W. Chen, “λ Domain Optimal Bit Allocation Algorithm

[0365] for High Efficiency Video Coding,” IEEE Transactions on Circuits and Systems for Video Technology, Vol. 28, No. 1, pp. 130–142, January 2018, doi: 10.1109 / TCSVT.2016.2598672.

[0366]

[41] Abdelaziz Djelouah, Joaquim Campos, Simone Schaub-Meyer and Christopher

[0367] Schroers, Neural inter-frame compression for video coding, ICCV, pages 6421–6429,

[0368] October 2019.

[0369]

[42] F. Bossen, Common Test Conditions and Software Reference Configurations, document Rec. JCTVC-J1100, Stockholm, Sweden, July 2012.

[0370]

[43] N.Kamaci, Y.Altunbasak and RMMersereau’s “Frame bit allocation for the

[0371] H.264 / AVC video coder via cauchy-density-based rate and distortionmodels,” IEEE Transactions on Circuits and Systems for Video Technology, Vol. 15, No. 8, pp. 994–1006, 2005.

[0372]

[44] S. Mallat and F. Falzon, “Analysis of low bit rate image transformcoding,” IEEE Transactions on Signal Processing, vol. 46, no. 4, pp. 1027–1042, 1998.

[0373] Fig.164000 is a block diagram illustrating an example video processing system 4000 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of system 4000. System 4000 may include an input 4002 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8 or 10 bit multi-component pixel values), or may be in a compressed or encoded format. Input 4002 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, a passive optical network (PON), and wireless interfaces such as Wi-Fi or a cellular interface.

[0374] System 4000 may include a codec component 4004 that can implement various codecs or encoding methods described in this document. Codec component 4004 can reduce the average bit rate of the video from input 4002 to the output of codec component 4004 to generate a codec representation of the video. Codec technology is therefore sometimes referred to as video compression or video transcoding technology. The output of codec component 4004 can be stored or sent via a communication connection as represented by component 4006. The bitstream (or codec) representation of the storage or communication transmission of the video received at input 4002 can be used by component 4008 to generate pixel values ​​or be transmitted to the displayable video of display interface 4010. The process of generating user-visible video from bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "codec" operations or tools, it will be understood that codec tools or operations are used at the encoder, and the corresponding decoding tools or operations of the inverse codec results will be performed by the decoder.

[0375] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), DisplayPort, etc. Examples of storage interfaces include Serial Advanced Technology Attachment (SATA), Peripheral Component Interconnect (PCI), Integrated Drive Electronics (IDE) interface, etc. The technology described in this document may be embodied in various electronic devices, such as mobile phones, laptops, smart phones, or other devices capable of performing digital data processing and / or video display.

[0376] Fig.174100. The apparatus 4100 may be used to implement one or more methods described herein. The apparatus 4100 may be embodied in a smartphone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. The apparatus 4100 may include one or more processors 4102, one or more memories 4104, and a video processing circuit 4106. The processor(s) 4102 may be configured to implement one or more methods described in this document. The memory(s) 4104 may be used to store data and code for implementing the methods and techniques described herein. The video processing circuit 4106 may be used to implement some of the techniques described in this document in a hardware circuit system. In some embodiments, the video processing circuit 4106 may be at least partially included in a processor 4102 (e.g., a graphics coprocessor).

[0377] Fig.18 4200. The method 4200 includes determining to apply a pre-processing function to visual media data as part of an image compression framework at step 4202. At step 4204, performing a conversion between the visual media data and a bitstream based on the image compression framework. Depending on the example, the conversion of step 4204 may include encoding at an encoder or decoding at a decoder.

[0378] It should be noted that the method 4200 may be implemented in an apparatus for processing video data, the apparatus comprising a processor and a non-transitory memory having instructions thereon, such as the video encoder 4400, the video decoder 4500, and / or the encoder 4600. In this case, the instructions, when executed by the processor, cause the processor to perform the method 4200. In addition, the method 4200 may be performed by a non-transitory computer-readable medium including a computer program product for use by a video codec device. The computer program product includes computer executable instructions stored on a non-transitory computer-readable medium, such that when executed by the processor, the video codec device performs the method 4200.

[0379] Fig.19 43 is a block diagram illustrating an example video codec system 4300 that can utilize the techniques of the present disclosure. The video codec system 4300 may include a source device 4310 and a target device 4320. The source device 4310 generates encoded video data, where the source device 4310 may be referred to as a video encoding device. The target device 4320 may decode the encoded video data generated by the source device 4310, where the target device 4320 may be referred to as a video decoding device.

[0380] Source device 4310 may include video source 4312, video encoder 4314 and input / output (I / O) interface 4316. Video source 4312 may include a source, such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of these sources. Video data may include one or more pictures. Video encoder 4314 encodes video data from video source 4312 to generate a bit stream. The bit stream may include a bit sequence that forms a codec representation of video data. The bit stream may include a codec picture and related data. The codec picture is a codec representation of a picture. Related data may include sequence parameter sets, picture parameter sets and other syntax structures. I / O interface 4316 may include a modulator / demodulator (modem) and / or a transmitter. Encoded video data may be directly sent to target device 4320 via network 4330 via I / O interface 4316. Encoded video data may also be stored on storage medium / server 4340 for access by target device 4320.

[0381] The target device 4320 may include an I / O interface 4326, a video decoder 4324, and a display device 4322. The I / O interface 4326 may include a receiver and / or a modem. The I / O interface 4326 may obtain coded video data from the source device 4310 or the storage medium / server 4340. The video decoder 4324 may decode the coded video data. The display device 4322 may display the decoded video data to the user. The display device 4322 may be integrated with the target device 4320, or may be external to the target device 4320 that may be configured to interface with an external display device.

[0382] The video encoder 4314 and the video decoder 4324 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other current and / or additional standards.

[0383] Fig. 20 is a block diagram showing an example of a video encoder 4400, which may be Fig.19 4300. Video encoder 4400 may be configured to perform any or all of the techniques of the present disclosure. Video encoder 4400 includes multiple functional components. The techniques described in this disclosure may be shared between various components of video encoder 4400. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0384] The functional components of the video encoder 4400 may include a segmentation unit 4401, a prediction unit 4402 (which may include a mode selection unit 4403, a motion estimation unit 4404, a motion compensation unit 4405 and an intra-frame prediction unit 4406), a residual generation unit 4407, a transform processing unit 4408, a quantization unit 4409, an inverse quantization unit 4410, an inverse transform unit 4411, a reconstruction unit 4412, a buffer 4413 and an entropy coding unit 4414.

[0385] In other examples, the video encoder 4400 may include more, fewer, or different functional components. In an example, the prediction unit 4402 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in IBC mode, where at least one reference picture is a picture in which the current video block is located.

[0386] Furthermore, some components, such as the motion estimation unit 4404 and the motion compensation unit 4405 , may be highly integrated, but are represented separately in the example of the video encoder 4400 for purposes of explanation.

[0387] The segmentation unit 4401 may segment the picture into one or more video blocks. The video encoder 4400 and the video decoder 4500 may support various video block sizes.

[0388] The mode selection unit 4403 may select one of the coding modes (e.g., intra or inter) based on the error result, and provide the resulting intra-coded block or inter-coded block to the residual generation unit 4407 to generate residual block data, and to the reconstruction unit 4412 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 4403 may select a combination of intra and inter prediction modes (CIIP), where the prediction is based on an inter prediction signal and an intra prediction signal. In the case of inter prediction, the mode selection unit 4403 may also select a resolution of the motion vector of the block (e.g., sub-pixel or integer pixel precision).

[0389] In order to perform inter-frame prediction on the current video block, the motion estimation unit 4404 may generate motion information of the current video block by comparing the current video block with one or more reference frames from the buffer 4413. The motion compensation unit 4405 may determine a predicted video block of the current video block based on the motion information and decoded samples of a picture from the buffer 4413 that is different from the picture associated with the current video block.

[0390] The motion estimation unit 4404 and the motion compensation unit 4405 may perform different operations on the current video block, eg, depending on whether the current video block is in an I slice, a P slice, or a B slice.

[0391] In some examples, the motion estimation unit 4404 may perform unidirectional prediction on the current video block, and the motion estimation unit 4404 may search the reference picture of list 0 or list 1 for the reference video block of the current video block. The motion estimation unit 4404 may then generate a reference index indicating the reference picture in list 0 or list 1 containing the reference video block and a motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 4404 may output the reference index, the prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 4405 may generate a predicted video block of the current block based on the reference video block indicated by the motion information of the current video block.

[0392] In other examples, the motion estimation unit 4404 may perform bidirectional prediction on the current video block, and the motion estimation unit 4404 may search for a reference video block of the current video block in the reference pictures in list 0, and may also search for another reference video block of the current video block in the reference pictures in list 1. The motion estimation unit 4404 may then generate a reference index indicating the reference pictures in list 0 and list 1 containing the reference video block and a motion vector indicating a spatial displacement between the reference video block and the current video block. The motion estimation unit 4404 may output the reference index and the motion vector of the current video block as the motion information of the current video block. The motion compensation unit 4405 may generate a predicted video block of the current video block based on the reference video block indicated by the motion information of the current video block.

[0393] In some examples, motion estimation unit 4404 may output a complete set of motion information for use in a decoding process by a decoder. In some examples, motion estimation unit 4404 may not output a complete set of motion information for the current video. Instead, motion estimation unit 4404 may reference motion information of another video block to signal motion information for the current video block. For example, motion estimation unit 4404 may determine that the motion information of the current video block is sufficiently similar to the motion information of an adjacent video block.

[0394] In one example, the motion estimation unit 4404 may indicate a value in a syntax structure associated with the current video block that indicates to the video decoder 4500 that the current video block has the same motion information as another video block.

[0395] In another example, the motion estimation unit 4404 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 4500 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0396] As discussed above, the video encoder 4400 may predictively signal motion vectors. Two examples of predictive signaling techniques that may be implemented by the video encoder 4400 include Advanced Motion Vector Prediction (AMVP) and Merge mode signaling.

[0397] The intra prediction unit 4406 may perform intra prediction on the current video block. When the intra prediction unit 4406 performs intra prediction on the current video block, the intra prediction unit 4406 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a predicted video block and various syntax elements.

[0398] The residual generation unit 4407 may generate residual data of the current video block by subtracting the predicted video block(s) of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample components of the samples in the current video block.

[0399] In other examples, such as in skip mode, there may be no residual data for the current video block and the residual generation unit 4407 may not perform a subtraction operation.

[0400] The transform processing unit 4408 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0401] After the transform processing unit 4408 generates a transform coefficient video block associated with the current video block, the quantization unit 4409 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0402] The inverse quantization unit 4410 and the inverse transform unit 4411 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 4412 may add the reconstructed residual video block to corresponding samples of one or more predicted video blocks generated by the prediction unit 4402 to generate a reconstructed video block associated with the current block for storage in the buffer 4413.

[0403] After the reconstruction unit 4412 reconstructs the video block, a loop filtering operation may be performed to reduce video block artifacts in the video block.

[0404] The entropy coding unit 4414 may receive data from other functional components of the video encoder 4400. When the entropy coding unit 4414 receives the data, the entropy coding unit 4414 may perform one or more entropy coding operations to generate entropy coded data and output a bitstream including the entropy coded data.

[0405] Fig.21 is a block diagram showing an example of a video decoder 4500, which may be Fig.19 4300. Video decoder 4500 may be configured to perform any or all of the techniques of the present disclosure. In the example shown, video decoder 4500 includes multiple functional components. The techniques described in this disclosure may be shared between various components of video decoder 4500. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0406] In the example shown, the video decoder 4500 includes an entropy decoding unit 4501, a motion compensation unit 4502, an intra prediction unit 4503, an inverse quantization unit 4504, an inverse transform unit 4505, a reconstruction unit 4506, and a buffer 4507. In some examples, the video decoder 4500 may perform a decoding process that is generally reciprocal to the encoding process described for the video encoder 4400.

[0407] The entropy decoding unit 4501 may retrieve a coded bitstream. The coded bitstream may include entropy-coded video data (e.g., coded blocks of video data). The entropy decoding unit 4501 may decode the entropy-coded video data, and from the entropy-decoded video data, the motion compensation unit 4502 may determine motion information including motion vectors, motion vector precision, reference picture list indexes, and other motion information. The motion compensation unit 4502 may determine such information, for example, by performing AMVP and Merge modes.

[0408] The motion compensation unit 4502 may generate a motion compensated block, and interpolation may be performed based on an interpolation filter. An identifier of an interpolation filter used with sub-pixel precision may be included in a syntax element.

[0409] The motion compensation unit 4502 may calculate interpolation of sub-integer pixels of a reference block using an interpolation filter as used by the video encoder 4400 during encoding of the video block. The motion compensation unit 4502 may determine the interpolation filter used by the video encoder 4400 based on received syntax information and use the interpolation filter to generate a prediction block.

[0410] The motion compensation unit 4502 can use some syntax information to determine the size of the blocks used to encode (multiple) frames and / or (multiple) slices of the encoded video sequence, partitioning information describing how each macroblock of the picture of the encoded video sequence is partitioned, a mode indicating how to encode each partition, one or more reference frames (and reference frame lists) for each inter-frame codec block, and other information used to decode the encoded video sequence.

[0411] The intra prediction unit 4503 may form a prediction block from spatially adjacent blocks using, for example, an intra prediction mode received in the bitstream. The inverse quantization unit 4504 inversely quantizes, i.e., dequantizes, the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 4501. The inverse transform unit 4505 applies an inverse transform.

[0412] The reconstruction unit 4506 may add the residual block to the corresponding prediction block generated by the motion compensation unit 4502 or the intra prediction unit 4503 to form a decoded block. If necessary, a deblocking filter may also be applied to filter the decoded block to remove block artifacts. The decoded video block is then stored in a buffer 4507, which provides reference blocks for subsequent motion compensation / intra prediction and also produces a decoded video for presentation on a display device.

[0413] Fig. 22 is a schematic diagram of an example encoder 4600. The encoder 4600 is suitable for implementing VVC technology. The encoder 4600 includes three loop filters, namely a deblocking filter (DF) 4602, a sample adaptive offset (SAO) 4604, and an adaptive loop filter (ALF) 4606. Unlike the DF 4602 that uses a predefined filter, the SAO 4604 and the ALF 4606 use the original samples of the current picture to reduce the mean square error between the original samples and the reconstructed samples by adding an offset and applying a finite impulse response (FIR) filter, respectively, using the auxiliary information of the codec to signal the offset and filter coefficients. The ALF 4606 is located at the last processing stage of each picture and can be regarded as a tool that attempts to capture and repair artifacts created by previous stages.

[0414] The encoder 4600 also includes an intra prediction component 4608 and a motion estimation / compensation (ME / MC) component 4610 configured to receive an input video. The intra prediction component 4608 is configured to perform intra prediction, while the ME / MC component 4610 is configured to perform inter prediction using a reference picture obtained from a reference picture buffer 4612. The residual block from the inter prediction or intra prediction is fed to a transform (T) component 4614 and a quantization (Q) component 4616 to generate quantized residual transform coefficients, which are fed to an entropy coding component 4618. The entropy coding component 4618 entropy codes and decodes the prediction result and the quantized transform coefficients and sends them to a video decoder (not shown). The quantized component output from the quantization component 4616 can be fed to an inverse quantization (IQ) component 4620, an inverse transform component 4622, and a reconstruction (REC) component 4624. The REC component 4624 is capable of outputting images to the DF 4602 , SAO 4604 , and ALF 4606 for filtering before these pictures are stored in the reference picture buffer 4612 .

[0415] Fig.23 4700. The method 4700 includes obtaining quantized residual potential samples at step 4702. At step 4704, performing a first process on the quantized residual potential samples. At step 4706, obtaining reconstructed potential samples based on the processed quantized residual potential samples. Depending on the example, step 4706 may include encoding at an encoder or decoding at a decoder.

[0416] It should be noted that method 4700 may be implemented in an apparatus for processing video data, the apparatus comprising a processor and a non-transitory memory having instructions thereon, such as video encoder 4400, video decoder 4500, and / or encoder 4600. In this case, the instructions, when executed by the processor, cause the processor to perform method 4700. In addition, method 4700 may be performed by a non-transitory computer-readable medium comprising a computer program product for use by a video codec device. The computer program product comprises computer executable instructions stored on a non-transitory computer-readable medium, such that when executed by a processor, the video codec device performs method 4700.

[0417] Fig.24 4800 is a flow chart of an example method 4800 of video processing. The method 4800 includes obtaining residual potential samples at step 4802. At step 4804, performing a second process on the residual potential samples. At step 4806, quantizing the processed residual potential samples to obtain quantized residual potential samples. Depending on the example, step 4706 may be included as part of encoding at an encoder or decoding at a decoder.

[0418] It should be noted that method 4800 may be implemented in an apparatus for processing video data, the apparatus comprising a processor and a non-transitory memory having instructions thereon, such as video encoder 4400, video decoder 4500, and / or encoder 4600. In this case, the instructions, when executed by the processor, cause the processor to perform method 4800. In addition, method 4800 may be performed by a non-transitory computer-readable medium comprising a computer program product for use by a video codec device. The computer program product comprises computer executable instructions stored on a non-transitory computer-readable medium, such that when executed by a processor, the video codec device performs method 4800.

[0419] Next a list of preferred solutions with some examples is provided.

[0420] The following solutions illustrate examples of the techniques discussed herein.

[0421] 1. An image decoding method, comprising the steps of: using an arithmetic decoder to obtain a reconstruction potential value The reconstruction potential value is fed into the synthetic neural network; based on the decoding parameters for the patch segmentation, the output feature map is patch-segmented into multiple parts at one or more locations; each part is fed separately into the next stage of the convolution layer to obtain the output spatial segmentation feature map; the spatial segmentation feature map is cropped and spatially stitched back to the complete feature map; continue until the image is reconstructed.

[0422] 2. An image encoding method, comprising the following steps: obtaining quantized potential values ​​and slice segmentation parameters; and encoding the potential values ​​and segmentation parameters into a bit stream.

[0423] 3. A device for processing video data, comprising: a processor; and a non-temporary memory having instructions thereon, wherein when the instructions are executed by the processor, the processor is caused to perform the method of any one of solutions 1 to 2.

[0424] 4. A non-transitory computer-readable medium comprising a computer program product for use by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, such that when the instructions are executed by a processor, the video codec device performs a method according to any one of solutions 1 to 2.

[0425] 5. A non-transitory computer-readable recording medium storing a bit stream of a video generated by a method performed by a video processing device, wherein the method includes the method of any one of solutions 1 to 2.

[0426] 6. A method for storing a bit stream of a video, comprising the method of any one of solutions 1 to 2.

[0427] 7. The methods, apparatus or systems described in this document.

[0428] Another list of preferred solutions with some examples is provided next.

[0429] 1. A method for processing video data in a neural network, comprising: obtaining a quantized residual potential sample point Processing quantization residual potential samples To obtain the potential sample points of the quantized residual after processing Quantize the potential sample points of the residual Add to the predicted sample (μ) to get the potential sample Based on potential sample points To perform conversion between visual media data and bitstreams.

[0430] 2. The method according to solution 1, further comprising: subtracting the predicted sample (μ) from the potential sample (y) to obtain a residual potential sample (w); and processing the residual potential sample w to obtain a processed residual sample (ws), wherein the processed residual sample ws is quantized to obtain a quantized residual potential sample

[0431] 3. A method according to any one of solutions 1 to 2, wherein the residual potential sample w is processed by a gain unit (T), and wherein the quantized residual potential sample w is Processed by the inverse gain unit (R).

[0432] 4. The method according to any one of solutions 1 to 3, wherein the inverse gain unit R applies the opposite function of the gain unit T.

[0433] 5. A method according to any one of solutions 1 to 4, wherein the inverse gain unit R reduces the potential sample points of the quantized residual The magnitude of , and wherein the gain unit T increases the magnitude of the residual potential sample w.

[0434] 6. The method according to any one of solutions 1 to 5, wherein the inverse gain unit R is implemented using a sample-by-sample multiplication according to:

[0435]

[0436] in is the residual potential sample indicated by index i, x, and y T is the processing vector, T[i] is the sample point of the processing vector, and is the quantized residual potential sample indicated by index i, x and y

[0437] 7. The method according to any one of solutions 1 to 6, wherein the inverse gain unit R is implemented using a sample-by-sample multiplication according to:

[0438]

[0439] in is the residual potential sample indicated by index i, x, and y T[K][i] is the sample point i in the processing vector T[K], and is the quantized residual potential sample indicated by index i, x and y

[0440] 8. A method according to any of solutions 1 to 7, wherein i is an index corresponding to the channel dimension.

[0441] 9. A method according to any one of solutions 1 to 8, wherein x or y is an index corresponding to a horizontal spatial dimension or a vertical spatial dimension.

[0442] 10. The method according to any one of solutions 1 to 9, wherein the gain unit R is implemented using a sample-by-sample multiplication according to:

[0443] ws[i,x,y]=T[K][i]×w[i,x,y]

[0444] where ws[i,x,y] is the processed residual potential sample ws indicated by indices i, x, and y, T[K][i] is the sample i in the processed vector T[K], and w[i,x,y] is the residual potential sample w indicated by indices i, x, and y.

[0445] 11. The method according to any of solutions 1 to 10, wherein the bitstream comprises at least two indications indicating the indices of the first processing vector T1 and the second processing vector T2 used in T[K].

[0446] 12. The method according to any of solutions 1 to 11, wherein K indicates the index of the processing vector.

[0447] 13. A method according to any one of solutions 1 to 12, wherein the potential sample point is divided into at least 2 patches, where a patch is a rectangular partition of potential samples, where the first processing vector T1 is used to obtain The sample point is then used to obtain the corresponding to the first slice and wherein the second processing vector T2 is used to obtain The sample points are then used to obtain the corresponding second slice Sample points.

[0448] 14. The method according to any of solutions 1 to 13, wherein vector T1 is different from vector T2.

[0449] 15. The method according to any of solutions 1 to 14, wherein a sigma scaling unit (P) is applied to the probability parameter (σ).

[0450] 16. The method according to any one of solutions 1 to 15, wherein a probability parameter (σ) is used to obtain quantized residual potential samples.

[0451] 17. The method according to any one of solutions 1 to 16, wherein the sigma scaling unit P is implemented using a sample-by-sample point multiplication according to:

[0452] σs[i,x,y]=P[K][i]×σ[i,x,y]

[0453] where σs[i,x,y] is the scaled probability parameter σs indicated by indices i, x, and y, P[K][i] is parameter i in the processing vector P[K], and σ[i,x,y] is the probability parameter σ indicated by indices i, x, and y.

[0454] 18. A method according to any one of solutions 1 to 17, wherein the probability parameter σ is processed by a sigma scaling unit P, wherein the processed probability parameter σ is used to obtain the quantized residual potential sample point in Then it is processed by the second gain unit T to obtain the processed quantized residual potential sample points The processed quantized residual potential sample is then used to obtain the potential sample This produces a reconstructed image.

[0455] 19. The method according to any one of solutions 1 to 18, wherein the reconstructed image is obtained by processing the potential samples with a transform process.

[0456] 20. A method according to any one of solutions 1 to 19, wherein the transformation process is an inverse transformation or a synthetic transformation.

[0457] 21. A method according to any one of solutions 1 to 20, wherein an indication for selecting a processing vector T or a processing vector P is included in the bitstream, wherein the processing vector T modifies the quantized residual potential samples And the processing vector P modifies the probability parameter σ.

[0458] 22. A method according to any one of solutions 1 to 21, wherein the residual potential sample w is processed before quantization, and wherein the quantized residual potential sample w is processed before quantization. is included in the bitstream.

[0459] 23. A method according to any one of solutions 1 to 22, wherein R and T are used in series before and after quantization to adjust the amplitude of the residual samples, wherein the amount of information included in the bitstream is obtained, wherein when R reduces the amplitude of the residual samples, the bit rate required to encode the residual samples is reduced, wherein the size of the bitstream for encoding the residual samples is reduced, wherein the compression ratio is increased, wherein increasing the compression ratio reduces the quality of the compressed image, and wherein R is used to control the compression ratio.

[0460] 24. A method according to any one of solutions 1 to 23, wherein when R reduces the magnitude of the residual sample, T counteracts by increasing the strength of the residual sample, and wherein the potential sample The magnitude remains the same on average.

[0461] 25. A method according to any one of solutions 1 to 24, wherein firstly quantized residual samples are obtained from a bitstream Then Through T processing to obtain the processed quantized residual sample points Quantized residual samples after processing It is then added to the predicted sample to get the potential sample

[0462] 26. A method according to any one of solutions 1 to 25, wherein the process performed by R and T is a sample-by-sample point multiplication with a vector performed as follows:

[0463] O[i]=T[i]×v[i]

[0464] where the i-th element of the output O is obtained by multiplying the i-th element of the vectors T and v.

[0465] 27. The method according to any one of solutions 1 to 26, wherein the process performed by T is performed as follows:

[0466]

[0467] in is the quantized residual sample point, and T is the processing Vector.

[0468] 28. The method according to any one of solutions 1 to 27, wherein the process performed by R is performed as follows:

[0469] ws[i]=R[i]×w[i]

[0470] where w is the residual latent sample, and R is the vector of processing w.

[0471] 29. The method according to any one of solutions 1 to 28, wherein the process performed by T is performed as follows:

[0472]

[0473] The i-th element of the output is obtained by multiplying the i-th element of the vector T1 with the i-th element of the quantized residual sample and adding the i-th sample of the vector T2.

[0474] 30. A method according to any one of solutions 1 to 29, wherein the sample points of the processed quantized residual sample points can be obtained according to the following:

[0475]

[0476] Where K is an index value that determines the index of the processing vector T[K] used to process the quantized residual sample, and wherein the value of K is used to determine which vector from a set of possible vectors is used to process the quantized residual sample.

[0477] 31. A method according to any one of solutions 1 to 30, wherein the sample of the processed quantized residual samples whose positions are given by the three indices i, x and y is obtained by The processing vector T[K] is multiplied to obtain, or wherein at least the first sample point of the processed quantized residual sample points is obtained according to the first vector T[K]. And obtain at least a second sample point according to the second vector T[M] or among them The samples are then used to obtain potential samples The latent samples are then processed through a transform to obtain a reconstructed image.

[0478] 32. A method according to any one of solutions 1 to 31, wherein T1 and T2 are used to process quantized residual samples. To obtain It is then added to the predicted sample μ to obtain the potential sample And the latent samples are then processed through the transform to obtain the reconstructed image.

[0479] 33. The method according to any of the solutions 1 to 32, wherein two indications are included in the bitstream to indicate which processing vector corresponds to the first slice and which vector corresponds to the second slice.

[0480] 34. A method according to any one of solutions 1 to 33, wherein applying P to probability parameters and applying T to quantized residual samples is performed so that the modification of the quantized residual samples results in different statistical characteristics, and wherein the probability parameters are modified by P to align the modification of the quantized residual samples.

[0481] 35. A method according to any of solutions 1 to 34, wherein a single index value is used to determine which vector P and which vector T to use.

[0482] 36. A method for processing video data in a neural network, comprising: obtaining a quantized residual potential value using a bit stream and an output of a first sub-network The first sub-network is not autoregressive; the second sub-network and the quantized potential value are used The reconstructed sample points are used to obtain the predicted value mean; the quantized residual sample points are used and predict mean to reconstruct the quantitative potential value The next sample point of ; and the use of quantized potential value and synthesize the transforms to obtain the reconstructed image.

[0483] 37. A method for processing video data in a neural network, comprising: transforming an input image using an analysis transform to obtain a potential value y; using a second network and a quantized potential value The reconstructed sample points are used to obtain the predicted value mean; the sample points of the potential value are subtracted from the predicted value to obtain the residual potential sample point w; the residual potential sample point Quantify and add to the predicted value to obtain the quantitative potential value and using the first subnetwork and quantized residual potential samples The bit stream is obtained by sampling points, where the first sub-network is not autoregressive.

[0484] 38. A method according to any one of solutions 36 to 37, wherein the first sub-network takes as input the first quantized super-potential value and generates a probability parameter.

[0485] 39. A method according to any of solutions 36 to 38, wherein obtaining samples of quantized residual potential values ​​comprises entropy decoding, wherein probability parameters and a bitstream are used as input.

[0486] 40. A method according to any of solutions 36 to 39, wherein obtaining the bitstream comprises entropy coding, wherein probability parameters and quantized residual potential values ​​are input.

[0487] 41. A method according to any of solutions 36 to 40, wherein the probability parameter does not include a mean value.

[0488] 42. A method according to any of solutions 36 to 41, wherein a zero mean probability distribution is used in entropy encoding or entropy decoding.

[0489] 43. A method according to any of solutions 36 to 42, wherein the second sub-network takes as input, in addition to the reconstructed samples of the quantized potential value, a second quantized super potential value.

[0490] 44. The method according to any one of solutions 36 to 43, wherein the first quantified super potential value and the second quantified super potential value are the same.

[0491] 45. A method according to any of solutions 36 to 44, wherein the quantized super potential value is obtained from a bitstream in a decoder.

[0492] 46. ​​A method according to any one of solutions 36 to 45, wherein a subnetwork is used to quantize the potential value y from the potential value y or the potential value Obtain quantified super potential value.

[0493] 47. A method according to any of solutions 36 to 46, wherein the second sub-network is autoregressive.

[0494] 48. A method according to any of solutions 36 to 47, wherein the second sub-network includes a context module.

[0495] 49. A method according to any of solutions 36 to 48, wherein the second subnet includes a super decoder module.

[0496] 50. A device for processing video data, comprising: a processor; and a non-volatile memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method according to any one of solutions 1 to 49.

[0497] 51. A non-transitory computer-readable medium comprising a computer program product for use with a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, such that when the instructions are executed by a processor, the video codec device performs a method according to any one of Solutions 1 to 49.

[0498] 52. A non-transitory computer-readable recording medium storing a bit stream of a video, the video being generated by a method performed by a video processing device, wherein the method comprises: obtaining a quantized residual potential sample point Processing quantization residual potential samples To obtain the potential sample points of the quantized residual after processing Quantize the potential sample points of the residual Add to the predicted sample (μ) to determine the potential sample and generating a bitstream based on the determination.

[0499] 53. A method for storing a bit stream of a video, comprising: obtaining a quantized residual potential sample point Processing quantization residual potential samples To obtain the potential sample points of the quantized residual after processing Quantize the potential sample points of the residual Add to the predicted sample (μ) to determine the potential sample generating a bitstream based on the determination; and storing the bitstream in a non-transitory computer-readable recording medium.

[0500] In the solution described herein, an encoder may comply with the format rules by generating a codec representation according to the format rules. In the solution described herein, a decoder may parse the syntax elements in the codec representation using the format rules to generate decoded video, knowing the presence and absence of the syntax elements according to the format rules.

[0501] In this document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, during the conversion from a pixel representation of a video to a corresponding bitstream representation, a video compression algorithm may be applied, and vice versa. As defined by the syntax, the bitstream representation of the current video block may correspond, for example, to bits that are juxtaposed or interspersed in different places within the bitstream. For example, macroblocks may be encoded according to error residual values ​​of transforms and codecs and also using bits in headers and other fields in the bitstream. In addition, during conversion, the decoder may parse the bitstream based on the determination, knowing that some fields may or may not be present, as described in the above solution. Similarly, the encoder may determine whether to include or not include certain syntax fields, and generate the codec representation accordingly by including or excluding syntax fields from the codec representation.

[0502] The disclosed and other solutions, examples, embodiments, modules and functional operations described in this document can be implemented in digital electronic circuits, or in computer software, firmware or hardware (including the structures disclosed in this document and their structural equivalents), or in a combination of one or more of them. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium, which are used to be executed by a data processing device or control the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances that affect machine-readable propagation signals, or a combination of one or more of them. The term "data processing device" includes all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal generated to encode information for transmission to a suitable receiver device, such as a machine-generated electrical signal, an optical signal, or an electromagnetic signal.

[0503] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language (including compiled or interpreted languages), and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0504] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits (e.g., field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs)), and the apparatus can also be implemented as special purpose logic circuits (e.g., field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs)).

[0505] Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or operably coupled to receive data from or transfer data to or from the one or more mass storage devices. However, a computer does not require such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and compact disk read-only memory (CD ROM) and digital versatile disk read-only memory (DVD-ROM) disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0506] Although this patent document contains many details, these details should not be interpreted as limitations on any subject matter or the scope of possible protection, but rather as descriptions of features of specific embodiments specified for specific technologies. Certain features described in the context of separate embodiments in this patent document may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be excluded from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0507] Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed, to achieve the desired results. In addition, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0508] Only a few implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and shown in this patent document.

[0509] A first component is directly coupled to a second component when there is no intermediate component between the first component and the second component other than a line, a trace, or another medium. A first component is indirectly coupled to a second component when there is an intermediate component between the first component and the second component other than a line, a trace, or another medium. The term "coupled" and its variations include direct coupling and indirect coupling. The use of the term "approximately" is meant to include a range of ±10% of the subsequent numerical value unless otherwise stated.

[0510] Although several embodiments have been provided in the present disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are considered to be illustrative rather than restrictive, and are not intended to be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0511] In addition, without departing from the scope of the present disclosure, the techniques, systems, subsystems and methods described and illustrated as discrete or separate in various embodiments may be combined or integrated with other systems, modules, techniques or methods. Other items shown or discussed as coupled may be directly connected, or may be indirectly coupled or communicated through some interface, device or intermediate component, whether electrically, mechanically or otherwise. Other examples of changes, substitutions and modifications may be determined by those skilled in the art and may be made without departing from the spirit and scope of the present disclosure.

[0512] The solutions listed in this disclosure may be used to compress an image, compress a video, compress a portion of an image, or compress a portion of a video.

[0513] Additionally, the techniques, systems, subsystems, and methods described and illustrated in various embodiments may be used to compress an image, compress a video, compress a portion of an image, or compress a portion of a video.

Claims

1. A method for processing video data in a neural network, comprising: Obtaining potential sample points of quantized residuals; Performing a first process on the quantized residual potential sample points; as well as Reconstructed potential samples are obtained based on the processed quantized residual potential samples.

2. A method for processing video data in a neural network, comprising: Get residual potential sample points; performing a second process on the residual potential sample points; as well as The processed residual potential samples are quantized to obtain quantized residual potential samples.

3. The method according to claim 1, wherein: The method is used for decoding the video data.

4. The method according to claim 1, wherein: The method is used to encode the video data, and obtaining quantized residual potential samples includes: Get residual potential sample points; performing a second process on the residual potential sample points; The processed residual potential samples are quantized to obtain the quantized residual potential samples.

5. The method according to claim 1, 3 or 4, wherein: The first processing is performed by an inverse gain unit.

6. The method according to claim 2 or 4, wherein: The second processing is performed by a gain unit.

7. The method according to claim 4, wherein: The first process applies an inverse function of the second process.

8. The method according to any one of claims 1 to 7, wherein: The first process adjusts the amplitude of the quantized residual potential sample points; or wherein the second process adjusts the amplitude of the residual potential sample points.

9. The method according to any one of claims 1 to 8, wherein: The first processing is based on a processing vector T.

10. The method according to any one of claims 1 to 9, wherein: The processed quantized residual potential sample is obtained based on the product of the processing vector T and the quantized residual potential sample.

11. The method according to any one of claims 1 to 10, wherein: The first process is implemented according to the following: in, is the processed quantized residual sample indicated by the indices i, x, and y T is a processing vector, T[i] is a sample of the processing vector, and is the quantized residual potential sample indicated by index i, x and y 12. The method according to any one of claims 1 to 10, wherein: The first process is implemented according to the following: in, is the processed quantized residual sample indicated by the indices i, x, and y T[K] is the processing vector, and is the quantized residual potential sample indicated by index i, x and y 13. The method according to any one of claims 1 to 12, wherein: The second processing is based on a processing vector R.

14. The method according to any one of claims 1 to 13, wherein: The processed residual potential sample is obtained based on the product of R and the residual potential sample.

15. The method according to any one of claims 1 to 14, wherein: The processing vector T used in the first processing is related to the vector R used in the second processing.

16. The method according to any one of claims 2 to 18, wherein: The second process is implemented according to the following: ws[i,x,y]=R[K][i]×w[i,x,y] Wherein ws[i,x,y] is the processed residual potential sample indicated by indexes i, x and y, R[K] is the processing vector, R[i] is the sample of the processing vector, and w[i,x,y] is the residual potential sample w indicated by indexes i, x and y.

17. The method according to any one of claims 2 to 16, wherein: The second process is implemented according to the following: ws[i,x,y]=R[i]×w[i,x,y] Wherein ws[i,x,y] is the processed residual potential sample indicated by index i, x and y, R is the processing vector, R[i] is the sample of the processing vector, and w[i,x,y] is the residual potential sample w indicated by index i, x and y.

18. The method according to any one of claims 11 to 17, wherein: i is the index corresponding to the channel dimension.

19. The method according to any one of claims 11 to 19, wherein: x or y is an index corresponding to the horizontal spatial dimension or the vertical spatial dimension.

20. The method according to any one of claims 11 to 19, wherein: K indicates the index of the vector.

21. The method according to claims 11 to 20, wherein: T=1 / R.

22. The method according to any one of claims 1 to 21, wherein: Different vectors in the first process are used for the first sample point and the second sample point, respectively.

23. The method according to claim 22, wherein: The different vectors are respectively based on different indications in the bitstream.

24. The method according to claim 22, wherein: The first sample point and the second sample point belong to different patches.

25. The method according to claim 24, wherein: The first sample point and the second sample point belong to different components.

26. The method according to any one of claims 1 to 25, wherein: Different vectors in the second process are used for the first sample point and the second sample point, respectively.

27. The method according to claim 26, wherein: The different vectors are respectively based on different indications in the bitstream.

28. The method according to claim 26, wherein: The first sample point and the second sample point belong to different patches.

29. The method of claim 22, wherein: The first sample point and the second sample point belong to different components.

30. The method according to any one of claims 1 to 29, wherein: The potential samples in the video data are divided into at least two slices.

31. The method according to claim 30, wherein: A patch is a rectangular partition of the potential samples.

32. The method according to any one of claims 1 to 31, wherein the potential samples are processed by a transformation process to obtain a reconstructed image.

33. The method of claim 32, wherein: The transform process is an inverse transform or a composite transform.

34. The method according to claims 1 to 33, wherein: The probability parameters are used in the entropy decoding process to obtain the potential samples of the quantized residual.

35. The method of claim 33, wherein: The first processing is performed after obtaining the quantized residual potential samples using the probability parameters.

36. The method according to claim 35 or 36, wherein: The probability parameters are obtained by scaling the initial probability parameters using the processing vector P.

37. The method of claim 34, wherein: The probability parameters are obtained according to: σs[i,x,y]=P[K][i]×σ[i,x,y] where σs[i,x,y] is the scaled probability parameter σs indicated by index i, x and y, P[i] is parameter i in the processing vector P, and σ[i,x,y] is the probability parameter σ indicated by index i, x and y.

38. The method of claim 34, wherein: The probabilities are obtained as follows: σs[i,x,y]=P[K][i]×σ[i,x,y] where σs[i,x,y] is the scaled probability parameter σs indicated by indices i, x and y, P[K][i] is parameter i in the processing vector P[K], and σ[i,x,y] is the probability parameter σ indicated by indices i, x and y.

39. The method according to any one of claims 34 to 38, wherein: i is the index corresponding to the channel dimension.

40. The method according to any one of claims 34 to 39, wherein: x or y is an index corresponding to the horizontal spatial dimension or the vertical spatial dimension.

41. The method according to any one of claims 34 to 39, wherein: K indicates the index of the vector.

42. The method according to claim 2 or 3, wherein: The residual potential sample is obtained by subtracting the predicted sample from the potential sample.

43. The method of claim 1, wherein: The processed quantized residual potential sample point is added to the predicted sample point to obtain the potential sample point.

44. An apparatus for processing video data, comprising: processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 43.

45. A non-transitory computer-readable medium, comprising a computer program product for use by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, such that the computer executable instructions, when executed by a processor, cause the video codec device to perform a method according to any one of claims 1 to 43.

46. ​​A non-transitory computer-readable recording medium storing a bit stream of a video generated by the method according to any one of claims 1 to 43.

Images