Adaptive image and video compression method based on neural network

Abstract

An image decoding method includes transforming an input image into potential sample points using analytical transformation; quantizing the potential sample points by using a super encoder to generate quantized super potential sample points; encoding the quantized superpotential sample points into a bit stream by using entropy coding; applying a potential sample point prediction process to obtain quantized potential sample points and quantized residual potential sample points based on the potential sample points using the quantized superpotential sample points; obtaining a prediction sample point after the potential sample point prediction process; and entropy encoding the quantized superpotential sample points and the quantized residual potential sample points into a bitstream.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims priority to U.S. Provisional Patent Application No. 63 / 390,263, filed on July 18, 2022, the teachings and disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] This patent application relates to the generation, storage and consumption of digital audio-visual media information in file format. Background Art

[0004] Digital video consumes the largest amount of bandwidth used on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, the demand for bandwidth used by digital video is likely to continue to grow. Summary of the invention

[0005] The disclosed aspects / embodiments provide techniques related to a neural network based adaptive image and video compression method. The present disclosure is directed to the insufficient memory problem during decoding when an image or video sequence is too large to fit into memory, resulting in decoding failure. The present disclosure provides a tiled partitioning scheme that provides the feasibility of successfully decoding from a bitstream regardless of spatial size, which is particularly beneficial for limited memory budgets or large resolution images / videos.

[0006] The first aspect relates to an image decoding method, comprising: performing an entropy decoding process to obtain quantized super potential samples and quantized residual potential samples Apply the potential sample prediction process to quantize the super potential samples and quantized residual potential samples Get quantified potential samples and applying a synthetic transformation process to use quantized latent samples Generate a reconstructed image.

[0007] Optionally, in any of the preceding aspects, another implementation of this aspect provides: receiving a bitstream including a header, wherein the header includes a model identifier (model_id), a metric and / or a quality, wherein the metric specifies a model used in the conversion, and the quality specifies a quality of a pre-trained model quality.

[0008] Optionally, in any of the preceding aspects, another implementation of this aspect provides that: the header specifies the height of the output picture (original_size_h) in number of luma samples, and / or the header specifies the width of the output picture (original_size_w) in number of luma samples.

[0009] Optionally, in any of the preceding aspects, another implementation of this aspect provides: the header specifies the height (resized_size_h) of the reconstructed picture after the synthesis transformation and before the resampling process in terms of the number of luma samples and / or the header specifies the width (resized_size_w) of the reconstructed picture after the synthesis transformation and before the resampling process in terms of the number of luma samples.

[0010] Optionally, in any of the preceding aspects, another implementation of this aspect provides that: the header specifies a height of the quantized residual latent value (latent_code_shape_h) and / or a width of the quantized residual latent value (latent_code_shape_w).

[0011] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: the header specifies an output bit depth (output_bit_depth) of the output reconstructed picture and / or a number of bits required to be shifted (output_bit_shift) when obtaining the output reconstructed picture.

[0012] Optionally, in any of the preceding aspects, another implementation of this aspect provides: the header specifies a double precision processing flag (double_precision_processing_flag), the double precision processing flag specifying whether double precision processing is enabled.

[0013] Optionally, in any of the preceding aspects, another implementation of this aspect provides that: the header specifies whether deterministic processing is applied when performing conversion between the visual media data and the bitstream.

[0014] Optionally, in any of the preceding aspects, another implementation of this aspect provides: the header specifies a fast resize flag (fast_resize_flag), the fast resize flag specifying whether to use fast resize.

[0015] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: performing a resampling process based on a fast resizing flag.

[0016] Optionally, in any of the preceding aspects, another implementation of this aspect provides that: the header specifies a number (num_second_level_tile or num_first_level_tile) for specifying the number of tiles.

[0017] Optionally, in any of the preceding aspects, another implementation of this aspect provides that: the number specifies a first level tile (num_first_level_tile) and / or a second level tile (num_second_level_tile).

[0018] Optionally, in any of the preceding aspects, another implementation of this aspect provides: performing a composition transform or a portion of a composition transform based on the number of slices.

[0019] Optionally, in any preceding aspect, another implementation of this aspect provides that: the header specifies the number of threads (num_wavefront_max or num_wavefront_min) used in wavefront processing.

[0020] Optionally, in any of the preceding aspects, another implementation of this aspect provides that: the header specifies a maximum number of threads used in wavefront processing (num_wavefront_max) and / or a minimum number of threads used in wavefront processing (num_wavefront_min).

[0021] Optionally, in any of the preceding aspects, another implementation of this aspect provides that: the header specifies the number of samples in each row that are shifted compared to the samples in the previous row (waveshift).

[0022] Optionally, in any of the preceding aspects, another implementation of this aspect provides that: the header specifies the number of parameter sets or filters used in the adaptive quantization process to control residual quantization.

[0023] Optionally, in any of the preceding aspects, another implementation of this aspect provides that: the header includes a parameter specifying how many times the adaptive quantization process is performed.

[0024] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: the adaptive quantization process is to modify the residual sample point and / or variance sample (σ) process.

[0025] Optionally, in any of the preceding aspects, another implementation of this aspect provides that: the header specifies the number of filters or parameter sets used in the residual sample skipping process.

[0026] Optionally, in any of the foregoing aspects, another implementation of this aspect provides: the header specifies the number of parameter sets used in the latent domain masking and scaling to determine the latent samples in the quantization Scaling at the decoder after being reconstructed.

[0027] Optionally, in any of the preceding aspects, another implementation of this aspect provides: the header specifies the number of parameter sets used in the latent domain masking and scaling to modify the quantized latent samples before applying the synthesis transform

[0028] Optionally, in any of the preceding aspects, another implementation of this aspect provides that: the header specifies whether a threshold operation greater than or less than a threshold is applied during adaptive quantization.

[0029] Optionally, in any of the preceding aspects, another implementation of this aspect provides that: the header specifies a value of a multiplier to be used in an adaptive quantization process or a sample skipping process or a potential scaling process prior to synthesis.

[0030] Optionally, in any of the preceding aspects, another implementation of this aspect provides that: the header specifies a value of a threshold to be used in an adaptive quantization process or a sample skipping process or a potential scaling process prior to synthesis.

[0031] Optionally, in any of the preceding aspects, another implementation of this aspect provides: the header includes a parameter that specifies a number of multipliers, thresholds, or greater than flags specified in the header.

[0032] Optionally, in any of the preceding aspects, another implementation of this aspect provides: the header specifies a number of parameter sets, wherein the parameter set includes a threshold parameter and a multiplier parameter.

[0033] Optionally, in any of the foregoing aspects, another implementation of this aspect provides: the header includes an adaptive offset enable flag, the adaptive offset enable flag specifying whether to use adaptive offset.

[0034] Optionally, in any of the preceding aspects, another implementation of this aspect provides that: the header specifies the number of horizontal partitions (num_horizontal_split) in the adaptive offset process and the number of vertical partitions (num_vertical_split) in the adaptive quantization process.

[0035] Optionally, in any of the preceding aspects, another implementation of this aspect provides: the header specifies an offset precision (offsetPrecision), and wherein the plurality of adaptive offset coefficients are multiplied by the offset precision and rounded to the nearest integer before encoding.

[0036] Optionally, in any of the preceding aspects, another implementation of this aspect provides: the header specifies an offset precision (offsetPrecision), and wherein the adaptive offset coefficient is modified based on the offset precision.

[0037] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: performing an entropy decoding process, the entropy decoding process comprising parsing two independent bitstreams, and wherein a first bitstream of the two independent bitstreams is decoded using a fixed probability density model.

[0038] Optionally, in any of the foregoing aspects, another implementation of this aspect provides: using a discrete cumulative distribution function to analyze the quantized super potential sample points And use the super prior variance decoder to process the quantized super potential samples The hyper-prior variance decoder is a neural network (NN) based sub-network for generating the Gaussian variance σ.

[0039] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: applying arithmetic decoding to a second bit stream of the two independent bit streams to obtain quantized residual potential samples and assume a zero-mean Gaussian distribution

[0040] Optionally, in any of the foregoing aspects, another implementation of this aspect provides: for quantizing super potential samples An inverse transform operation is performed, and wherein the inverse transform operation is performed by a super a priori variance decoder.

[0041] Optionally, in any of the foregoing aspects, another implementation of this aspect provides: the output of the inverse transform operation is cascaded with the output of the context model module to generate a cascaded output, wherein the cascaded output is processed by the prediction fusion model to generate a prediction sample μ, and wherein the prediction sample is added to the quantized residual potential sample To obtain the potential sample points for quantification

[0042] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: the potential sample point prediction process is an autoregressive process.

[0043] Optionally, in any of the foregoing aspects, another implementation of this aspect provides: quantized potential samples in different rows are processed in parallel.

[0044] Optionally, in any of the preceding aspects, another implementation of this aspect provides: rounding the output of the super encoder.

[0045] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: using the Gaussian variance variable σ obtained as the output of the super-prior variance decoder to quantize the residual potential sample points Perform entropy coding.

[0046] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: encoder configuration parameters are pre-optimized.

[0047] Optionally, in any of the preceding aspects, another implementation of this aspect provides: the method is performed by an encoder, and wherein the prepare_weights() function of the encoder is configured to calculate default pre-optimized encoder configuration parameters.

[0048] Optionally, in any of the preceding aspects, another implementation of this aspect provides that: the write_weights() function of the encoder includes default pre-optimized encoder configuration parameters in the high-level syntax of the bitstream.

[0049] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides: not performing a rate-distortion optimization process.

[0050] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that: the decoding process is not performed as part of the encoding method.

[0051] Optionally, in any of the preceding aspects, another implementation of this aspect provides: using neural network-based adaptive image and video compression as disclosed herein.

[0052] A second aspect relates to an apparatus for processing video data, comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform any one of the disclosed methods.

[0053] A third aspect relates to a non-transitory computer-readable medium, comprising a computer program product used by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, so that when the computer executable instructions are executed by a processor, the video codec device performs any one of the disclosed methods.

[0054] A fourth aspect relates to a non-transitory computer-readable recording medium storing a bit stream of a video, wherein the bit stream of the video is generated by a method performed by a video processing device, wherein the method includes any one of the disclosed methods.

[0055] A fifth aspect relates to a method for storing a bitstream of a video, including the method of any one of the disclosed implementations.

[0056] A sixth aspect relates to a method, apparatus or system described in this document.

[0057] The seventh aspect relates to an image decoding method, comprising: performing an entropy decoding process to obtain a quantized super potential sample point and quantized residual potential samples Apply the potential sample prediction process to quantize the super potential samples and quantized residual potential samples Get quantified potential samples and applying a synthetic transformation process to use quantized latent samples Generate a reconstructed image.

[0058] The eighth aspect relates to an image encoding method, comprising: using an analysis transform to transform an input image into a potential sample y; using a super encoder to quantize the potential sample y to generate a quantized super potential sample Use entropy coding to convert the quantized super potential samples Encoded into a bitstream; applies a latent sample prediction process to use quantized super latent samples Get quantized potential samples based on potential samples y and quantized residual potential samples After the potential sample prediction process, the predicted sample μ is obtained; and the quantized super potential sample and quantized residual potential samples Entropy encoding into a bit stream.

[0059] Optionally, in any of the preceding aspects, another implementation of this aspect provides that: the header specifies the height of the original input picture before the resampling process (original_size_h) in number of luma samples, and the header specifies the width of the original input picture before the resampling process (original_size_w) in number of luma samples.

[0060] For purposes of clarity, any of the aforementioned embodiments may be combined with any one or more of the other aforementioned embodiments to create new embodiments within the scope of the present disclosure.

[0061]

[0011] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] For a more complete understanding of the present disclosure, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0063] Figure 1 An example of a typical transform coding scheme is illustrated.

[0064] Figure 2 An example of quantization potential values ​​when using a super encoder / decoder is illustrated.

[0065] Figure 3 An example of a network architecture for an autoencoder that implements a hyper-prior model is illustrated.

[0066] Figure 4 We illustrate a combined model that jointly optimizes an autoregressive component that estimates the probability distribution of latent values ​​from the causal context (i.e., the context model) and a hyper-prior and an underlying autoencoder.

[0067] Figure 5An encoding process using a super encoder and a super decoder is illustrated.

[0068] Figure 6 An example decoding process is illustrated.

[0069] Figure 7 Example implementations of encoding and decoding processes are illustrated.

[0070] Figure 8 A 2-dimensional positive wavelet transform is illustrated.

[0071] Fig. 9 A possible partitioning of the latent representation after a 2D forward transform is illustrated.

[0072] Figure 10A-10B A structural example of the proposed decoder architecture is illustrated.

[0073] Figure 11A-11B A structural example of the proposed encoder architecture is illustrated.

[0074] Fig.12 Details of attention blocks, residual units, and residual blocks are illustrated.

[0075] Fig.13 The details of the residual down-sampling block and the residual up-sampling block are illustrated.

[0076] Fig.14 The utilized masked convolution kernel is illustrated.

[0077] Fig.15 Potential sample processing modes according to wavefront parallel processing (WPP) are illustrated.

[0078] Fig.16 Examples of vertical and horizontal tiling of the synthesis transformation network are illustrated.

[0079] Fig.17 The structure of the discriminator is illustrated.

[0080] Fig.18 is a block diagram illustrating an example video processing system.

[0081] Fig.19 is a block diagram of an example video processing device.

[0082] Fig. 20 is a flow chart of an example video processing method.

[0083] Fig.21 is a block diagram illustrating an example video encoding and decoding system.

[0084] Fig. 22 is a block diagram illustrating an example encoder.

[0085] Fig.23is a block diagram illustrating an example decoder.

[0086] Fig.24 is a schematic diagram of an example encoder.

[0087] Fig.25 The present invention relates to an image decoding method according to an embodiment of the present disclosure.

[0088] Fig.26 It is an image encoding method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0089] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or yet to be developed. The present disclosure should not be limited in any way to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims and their full equivalents.

[0090] The section headings used in this document are for ease of understanding and do not limit the applicability of the techniques and embodiments disclosed in each section to only that section. In addition, the techniques described herein are applicable to other video codec protocols and designs.

[0091] 1. Summary.

[0092] A neural network-based image and video compression method includes an autoregressive subnetwork and an entropy coding and decoding engine, wherein the entropy coding and decoding are performed independently of the autoregressive subnetwork.

[0093] 2. Background.

[0094] The past decade has witnessed the rapid development of deep learning in various fields, especially in computer vision and image processing. Inspired by the great success of deep learning techniques in the field of computer vision, many researchers have shifted their attention from traditional image / video compression techniques to neural image / video compression techniques. Neural networks were originally invented as a result of interdisciplinary research in neuroscience and mathematics. They have shown strong capabilities in nonlinear transformation and classification. Neural network-based image / video compression techniques have made significant progress in the past five years. It is reported that the latest neural network-based image compression algorithms [1] achieve RD performance comparable to that of Versatile Video Codec (VVC) [2], which is the latest video codec standard developed by the Joint Video Experts Group (JVET) with experts from MPEG and VCEG. With the continuous improvement of neural image compression performance, neural network-based video compression has become an actively developing research field. However, due to the inherent difficulty of the problem, neural network-based video codecs are still in their infancy.

[0095] 2.1 Image / video compression.

[0096] Image / video compression generally refers to computing techniques that encode images / videos into binary codes for easy storage and transmission. The binary codes may or may not support lossless reconstruction of the original image / video, which are called lossless compression and lossy compression. Since lossless reconstruction is not necessary in most scenarios, most efforts are invested in lossy compression. The performance of image / video compression algorithms is usually evaluated from two aspects, namely compression ratio and reconstruction quality. The compression ratio is directly related to the number of binary codes, the less the better; the reconstruction quality is measured by comparing the reconstructed image / video with the original image / video, the higher the better.

[0097] Image / video compression techniques can be divided into two branches, classical video codec methods and neural network-based video compression methods. Classical video codec schemes adopt transform-based solutions, in which researchers exploit statistical dependencies in latent variables (e.g., DCT or wavelet coefficients) by carefully hand-designing entropy codes to simulate dependencies in the quantized domain. There are two styles of neural network-based video compression, neural network-based codec tools and end-to-end neural network-based video compression. The former is embedded into the existing classical video codec as a codec tool and is only part of the framework, while the latter is an independent framework developed based on neural networks and does not rely on classical video codecs.

[0098] Over the past three decades, a series of classic video codec standards have been developed to accommodate the growing visual content. The International Organization for Standardization ISO / IEC has two expert groups, the Joint Photographic Experts Group (JPEG) and the Moving Picture Experts Group (MPEG), and ITU-T also has its own Video Codec Experts Group (VCEG), which is used for the standardization of image / video codec technologies. Influential video codec standards released by these organizations include JPEG, JPEG 2000, H.262, H.264 / AVC, and H.265 / HEVC. Following H.265 / HEVC, the Joint Video Experts Group (JVET) composed of MPEG and VCEG has been working on a new video codec standard, the Versatile Video Codec (VVC). The first version of VVC was released in July 2020. Compared with HEVC, VVC reports an average bitrate reduction of 50% at the same visual quality.

[0099] Neural network-based image / video compression is not new, as many researchers are working on neural network-based image codecs [3]. However, the network architecture is relatively shallow and the performance is not satisfactory. Thanks to the support of abundant data and powerful computing resources, neural network-based methods have been better utilized in various applications. Currently, neural network-based image / video compression has shown promising improvements, proving its feasibility. However, this technology is far from mature and many challenges need to be addressed.

[0100] 2.2 Neural Network.

[0101] A neural network, also called an artificial neural network (ANN), is a computational model used in machine learning techniques that typically consists of multiple processing layers, each of which consists of multiple simple but nonlinear basic computational units. One advantage of such deep networks is believed to be the ability to process data with multiple levels of abstraction and transform the data into different types of representations. Note that these representations are not manually designed; instead, the deep network comprising (multiple) processing layers is learned from massive amounts of data using a general machine learning process. Deep learning eliminates the need for manual representations and is therefore considered particularly suitable for processing native unstructured data, such as acoustic and visual signals, which has been a long-standing problem in the field of artificial intelligence.

[0102] 2.3 Neural Networks for Image Compression.

[0103] Existing neural networks for image compression methods can be divided into two categories, namely pixel probability modeling and autoencoders. The former belongs to the predictive encoding and decoding strategy, while the latter is a transformation-based solution. Sometimes, these two methods are combined in the literature.

[0104] 2.3.1 Pixel probability modeling.

[0105] According to Shannon's information theory [6], the best lossless codec can achieve the minimum decoding rate, -log2p(x), where p(x) is the probability of symbol x. Many lossless codecs have been developed in the literature, among which arithmetic coding is considered to be one of the best methods [7]. Given a probability distribution p(x), arithmetic coding ensures that the coding rate is as close to its theoretical limit -log2p(x) as possible without considering round-off errors. Therefore, the remaining question is how to determine the probability, which is very challenging for natural images / videos due to the curse of dimensionality.

[0106] Following the predictive coding strategy, one way to model p(x) is to predict pixel probabilities one by one in raster scan order based on previous observations, where x is the image.

[0107] p(x)=p(x1)p(x2|x1)…p(x i |x1,…,x i-1 )…p(x m×n |x1,…,x m×n-1 ) (1) where m and n are the height and width of the image, respectively. The previous observation is also called the context of the current pixel. When the image is large, it may be difficult to estimate the conditional probability, so a simplified approach is to limit the scope of its context.

[0108] p(x)=p(x1)p(x2|x1)…p(x i |x i-k ,…,x i-1 )…p(x m×n |x m×n-k ,…,x m×n-1 ) (2)where k is a predefined constant that controls the scope of the context.

[0109] It should be noted that the condition may also take into account the sample values ​​of other color components. For example, when encoding and decoding RGB color components, the R sample depends on the previously encoded and decoded pixels (including R / G / B samples), the current G sample can be encoded and decoded based on the previously encoded and decoded pixels and the current R sample, and for the encoding and decoding of the current B sample, the previously encoded and decoded pixels and the current R and G samples can also be considered.

[0110] Most compression methods directly model the probability distribution in the pixel domain. Some researchers have also tried to model the probability distribution as a conditional probability distribution based on explicit or latent representations. That is, we can estimate

[0111]

[0112] Where h is an additional condition, and p(x) = p(h)p(x|h), which means that modeling can be divided into unconditional and conditional. The additional condition can be image label information or high-level representation.

[0113] 2.3.2 Autoencoder.

[0114] Autoencoders originated from the famous work by Hinton and Salakhutdinov

[17] . The method is trained for dimensionality reduction and consists of two parts: encoding and decoding. The encoding part converts the high-dimensional input signal into a low-dimensional representation, usually with reduced spatial size but more channels. The decoding part attempts to recover the high-dimensional input from the low-dimensional representation. Autoencoders achieve automatic learning of representations and eliminate the need for handcrafted features, which is also considered one of the most important advantages of neural networks.

[0115] Figure 1The following is an example of a typical transform coding scheme. The original image x is analyzed by the network g a is transformed to achieve the latent representation y. The latent representation y is quantized and compressed into bits. The number of bits R is used to measure the encoding and decoding rate. Then, the quantized latent representation By synthetic network g s Perform inverse transform to obtain the reconstructed image The distortion is calculated in the perceptual space by using the function g p Transform x and Calculated.

[0116] It is intuitive to apply autoencoder networks to lossy image compression. We just need to encode the latent representation learned from a trained neural network. However, adapting autoencoders to image compression is non-trivial because the original autoencoders are not optimized for compression, so directly using the trained autoencoders is not efficient. In addition, there are other significant challenges. First, the low-dimensional representation should be quantized before encoding, but quantization is not differentiable, which is required for backpropagation when training neural networks. Second, the goals in the compression scenario are different because both distortion and rate need to be considered. Estimating the rate is challenging. Third, practical image coding and decoding schemes need to support variable rates, scalability, encoding / decoding speed, and interoperability. To address these challenges, many researchers have been actively contributing to this field.

[0117] Prototype autoencoder for image compression in Figure 1 It can be regarded as a transform coding strategy. The original image x is transformed into y=g using the analysis network. a (x), where y is the latent representation that will be quantized and encoded. The synthesis network will transform the quantized latent representation Perform inverse transform to obtain the reconstructed image This framework utilizes the distortion loss function Training is performed, where D is the sum of x and The distortion between is the rate of computation or estimation, and λ is the Lagrange multiplier. It should be noted that D can be computed in the pixel domain or the perception domain. All existing research works follow this prototype, and the difference may only be the network structure or loss function.

[0118] In terms of network structure, recurrent neural networks (RNNs) and convolutional neural networks (CNNs) are the most widely used architectures. In the RNN-related category, Toderici et al.

[18] proposed a general framework for variable rate image compression using RNNs. They used binary quantization to generate codes and did not consider the rate during training. The framework does provide scalable encoding and decoding capabilities, and RNNs with convolutional and deconvolutional layers are reported to perform well. Toderici et al.

[19] subsequently proposed an improved version that compresses binary codes by upgrading the encoder using a neural network similar to PixelRNN. The performance on the Kodak image dataset using the MS-SSIM evaluation metric is reported to be better than JPEG. Johnston et al.

[20] further improved the RNN-based solution by introducing hidden state activation. In addition, an SSIM-weighted loss function was designed and a spatially adaptive bitrate mechanism was enabled. Using MS-SSIM as the evaluation metric, they achieved better results than BPG on the Kodak image dataset. Covell et al.

[21] supported spatially adaptive bitrate by training an RNN that tolerates stop codes.

[0119] Ballé et al.

[22] proposed a general framework for rate-distortion optimized image compression. They used multi-level quantization to generate integer codes and considered the rate during training, i.e., the loss is a joint rate-distortion cost, which can be MSE or other. They added random uniform noise during training to stimulate quantization and used the differential entropy of the noise code as a proxy for the rate. They used generalized divisive normalization (GDN) as the network structure, which includes a linear mapping and subsequent nonlinear parameter normalization. The effectiveness of GDN in image encoding and decoding was verified in

[23] . Ballé et al.

[24] then proposed an improved version, in which they used 3 convolutional layers, each followed by a downsampling layer and a GDN layer as the forward transform. Correspondingly, they used 3 inverse GDN layers, each followed by an upsampling layer and a convolutional layer to simulate the inverse transform. In addition, an arithmetic coding method was designed to compress integer codes. The performance in terms of MSE on the Kodak dataset is reported to be better than JPEG and JPEG 2000. In addition, Ballé et al.

[25] improved the method by designing a scale hyper-prior in the autoencoder. They use the subnetwork h a Convert the latent representation of y to z = h a (y), and z will be quantized and transmitted as auxiliary information. Correspondingly, the inverse transform is performed by sub-network h s The subnet h s Try to quantify the auxiliary information Decoded as quantized The standard deviation of is further used during arithmetic coding and decoding of . On the Kodak image set, their method is slightly worse than BPG in terms of PSNR. D.Minnen et al.

[26] further exploited the structure in the residual space by introducing an autoregressive model to estimate both the standard deviation and the mean. In the latest work

[27] , Z.Cheng et al. used a Gaussian mixture model to further eliminate the redundancy in the residual. The performance reported on the Kodak image set using PSNR as the evaluation metric is comparable to that of VVC

[28] .

[0120] 2.3.3 Super-prior model.

[0121] In the transform codec approach to image compression, the encoder subnetwork (Section 2.3.2) uses a parametric analysis transform The image vector x is transformed into a latent representation y, which is then quantized to form because It is a discrete value, so it can be losslessly compressed using entropy coding and decoding technology (such as arithmetic coding and decoding) and transmitted as a bit sequence.

[0122] from Figure 2 It can be clearly seen from the left and right images that There is a significant spatial dependency between the elements of . Notably, their scales (middle right image) appear to be spatially coupled. In

[25] , an additional set of random variables was introduced to capture spatial dependencies and further reduce redundancy. In this case, the image compression network Figure 3 Depicted in.

[0123] exist Figure 3 In the figure, the left side of the model is the encoder g a and decoder g s (Explained in Section 2.3.2). The right side is used to obtain The additional super encoder h a and super decoder h s In this architecture, the encoder processes the input image x through g a The response y is fed to h a The distribution of standard deviations in z is summarized in . z is then quantized The encoder then uses the quantized vector to estimate the spatial distribution of the standard deviation σ and use it to compress and transmit the quantized image representation The decoder first recovers the compressed signal Then, it uses h s to obtain σ, which provides it with the correct probability estimate for successful recovery Then, it will Feed to g s to obtain the reconstructed image.

[0124] When super encoders and super decoders are added to the image compression network, the potential value of quantization is reduced. of space redundancy. Figure 2 The rightmost image in the figure corresponds to the quantized potential values ​​when using the super encoder / decoder. Compared to the middle right image, the spatial redundancy is significantly reduced due to the lower sample correlation of the quantized potential values.

[0125] exist Figure 2 In Figure 1, an image from the Kodak dataset is shown on the left; a visualization of the latent representation y of the image is shown in the middle left; the standard deviation σ of the latent values ​​is shown in the middle right; and the latent values ​​y after being processed by the hyper-prior (super-encoder and decoder) networks are shown on the right.

[0126] Figure 3 Illustrated network architecture of an autoencoder implementing a super-prior model. An image of the autoencoder network is shown on the left, and the right corresponds to the super-prior subnetwork. The analysis transform and synthesis transform are denoted as g a and g s Q represents quantization, and AE and AD represent arithmetic encoder and arithmetic decoder respectively. The super prior model consists of two sub-networks, the super encoder (with h a denoted) and super decoder (with h s denoted by ). The hyper-prior model generates a quantified hyper-potential value It includes potential values ​​for quantization The probability distribution information of the sample points. is included in the bitstream and is related to Transmitted together to the receiver (decoder).

[0127] 2.3.4 Context model.

[0128] Although the hyper-prior model improves the potential value of quantification , but additional improvement can be obtained by leveraging autoregressive models (context models) that predict the quantified latent value from the causal context.

[0129] The term autoregressive means that the output of a process is later used as the input to that process. For example, the context model subnetwork generates a sample of a potential value that is later used as the input to get the next sample.

[0130] The authors in

[26] utilized a joint architecture where both the super-prior model sub-network (super-encoder and super-decoder) and the context model sub-network were utilized. The super-prior and context models were combined to learn the quantized latent value The probability model of is then used for entropy coding and decoding. Figure 4 As depicted in , the outputs of the context subnetwork and the super decoder subnetwork are combined by a subnetwork called the entropy parameters, which generates the mean μ and scale (or variance) σ parameters for the Gaussian probability model. The Gaussian probability model is then used to encode the quantized latent value samples into the bitstream with the help of the arithmetic encoder (AE) module. In the decoder, the Gaussian probability model is utilized by the arithmetic decoder (AD) module to obtain the quantized latent value from the bitstream.

[0131] Figure 4 The combined model of jointly optimizing an autoregressive component with a hyperprior and an underlying autoencoder is illustrated. The autoregressive component estimates the probability distribution of the latent value according to its causal context (context model). The real-valued latent representation is quantized (Q) to create a quantized latent value and quantified super potential They are compressed into a bitstream using an arithmetic encoder (AE) and decompressed by an arithmetic decoder (AD). The highlighted areas correspond to the components executed by the receiver (i.e., decoder) to recover the image from the compressed bitstream.

[0132] Typically, the latent samples are modeled as Gaussian distributions or Gaussian mixture models (but not limited to these). In

[26] , and according to Figure 4 , the context model and the hyper-prior are jointly used to estimate the probability distribution of the underlying sample points. Since the Gaussian distribution can be defined by the mean and variance (also called sigma or scale), a joint model is used to estimate the mean and variance (denoted as μ and σ).

[0133] 2.3.5 Encoding process using the joint autoregressive hyper-prior model.

[0134] Figure 4 This corresponds to the prior art compression method proposed in

[26] . In this section and the next, the encoding and decoding processes are described respectively.

[0135] Figure 5 The encoding process according to

[26] is illustrated.

[0136] exist Figure 5 The encoding process is depicted in Figure 1. The input image is first processed by the encoder subnetwork. The encoder transforms the input image into a transformed representation called a latent value, denoted by y. y is then input to the quantizer block denoted by Q to obtain a quantized latent value Then it is converted into a bit stream (bits1) using an arithmetic coding module (denoted as AE). The arithmetic coding block converts the Each sample point is converted into a bit stream (bits1).

[0137] The super encoder module, context module, super decoder module and entropy parameter sub-network are used to estimate the quantized latent value The potential value y is input to the super encoder, and the super encoder outputs the super potential value (denoted by z). The super potential value is then quantized The arithmetic encoding (AE) module is used to generate a second bit stream (bits2). The decomposition entropy module generates a probability distribution, which is used to encode the quantized super potential value into a bit stream. The quantized super potential value includes information about the quantized potential value. information about the probability distribution of .

[0138] The entropy parameter subnetwork generates a probability distribution estimate, which is used to quantize the potential value The information generated by the entropy parameters usually includes the mean μ and the scale (or variance) σ parameters, which are used together to obtain a Gaussian probability distribution. The Gaussian distribution of a random variable x is defined as where the parameter μ is the mean or expected value of the distribution (as well as its median and mode), and the parameter σ is its standard deviation (or variance, or scale). To define a Gaussian distribution, the mean and variance need to be determined. In

[26] , the entropy parameter module is used to estimate the mean and variance values.

[0139] The subnetwork superdecoder generates part of the information used by the entropy parameter subnetwork and another part of the information is generated by an autoregressive module called the context module. The context module uses the samples that have been encoded by the arithmetic coding (AE) module to generate information about the probability distribution of the samples of the quantized potential value. It is usually a matrix consisting of many sample points. Sample points can be indicated using indices, such as or Depends on the matrix The dimension of Encoded one by one by the AE, usually using raster scan order. In raster scan order, the rows of the matrix are processed from top to bottom, where the samples in the row are processed from left to right. In such a scenario (where the AE encodes the samples into the bitstream using raster scan order), the context module uses the previously encoded samples to generate the same samples in raster scan order. The information generated by the context module and the super decoder is combined by the entropy parameter module to generate a potential value for quantizing Encoded as a probability distribution of the bit stream (bits1).

[0140] Finally, the first and second bit streams are sent to a decoder as a result of the encoding process.

[0141] Note that other names may be used for the modules described above.

[0142] In the above description, Figure 5 All the elements in are collectively referred to as encoders. The analytical transformation that transforms the input image into a latent representation is also called an encoder (or autoencoder).

[0143] 2.3.6. Decoding process using the joint autoregressive hyper-prior model.

[0144] Figure 6 The decoding process corresponding to

[26] is illustrated. Figure 6 Depicted is the decoding process corresponding to

[26] .

[0145] During the decoding process, the decoder first receives the first bit stream (bits1) and the second bit stream (bits2) generated by the corresponding encoder. Bits2 is first decoded by the arithmetic decoding (AD) module by using the probability distribution generated by the decomposition entropy sub-network. The decomposition entropy module usually generates the probability distribution using a predetermined template, such as a predetermined mean and variance value in the case of a Gaussian distribution. The output of the arithmetic decoding process of bits2 is This is the quantized super potential value. The AD process will recover the AE process applied in the encoder. The AE and AD processes are lossless, which means that the quantized super potential value generated by the encoder Can be reconstructed at the decoder without any changes.

[0146] In obtaining Afterwards, it is processed by the super decoder, whose output is fed to the entropy parameter module. The three subnetworks employed in the decoder: context, super decoder, and entropy parameters are the same as those in the encoder. Therefore, the exact same probability distribution is available in the decoder (as in the encoder), which is essential for reconstructing the quantized latent values ​​without any loss. As a result, the quantized potential values ​​obtained in the encoder can be obtained in the decoder The same version of .

[0147] After obtaining the probability distribution (e.g., mean and variance parameters) by the entropy parameter subnetwork, the arithmetic decoding module decodes the quantized potential value samples one by one from the bitstream bits1. From a practical point of view, the autoregressive model (context model) is serial in nature and therefore cannot be accelerated using techniques such as parallelization.

[0148] Finally, the fully reconstructed quantized potential value is input to the composite transform (in Figure 6 The decoder module is used to obtain the reconstructed image.

[0149] In the above description, Figure 6 All elements in are collectively referred to as decoders. The synthetic transform that converts the quantized latent values ​​into the reconstructed image is also called a decoder (or auto-decoder).

[0150] 2.3.7. Wavelet-based neural compression architecture.

[0151] Figure 5 The analysis transform in (expressed as an encoder) and Figure 6 The synthetic transform in (denoted as a decoder) can be replaced by a wavelet-based transform. Figure 7 An example implementation of a wavelet based transform is illustrated. In the figure, first the input image is converted from RGB color format to YUV color format. This conversion process is optional and can be missing in other implementations. However, if such a conversion is applied at the input image, the inverse conversion (from YUV to RGB) is also applied before generating the output image. In addition, 2 additional post-processing modules (Post-processing 1 and 2) are shown in the figure. These modules are also optional and can be missing in other implementations. The core of the wavelet transform based encoder consists of a wavelet based forward transform, a quantization module, and an entropy encoding and decoding module. After applying these 3 modules to the input image, a bitstream is generated. The core of the decoding process consists of entropy decoding, a dequantization process, and a wavelet based inverse transform operation. The decoding process converts the bitstream into an output image. The encoding and decoding process is described below Figure 7 Depicted in.

[0152] After applying the wavelet-based forward transform to the input image, the image is divided into its frequency components in the output of the wavelet-based forward transform. The output of the two-dimensional (2D) forward wavelet transform (denoted as the iWave forward module in the figure) can be Figure 8 The input to the transformation is the image of the castle. In this example, an output with 7 distinct regions is obtained after the transformation. The number of distinct regions depends on the specific implementation of the transformation and may be different from 7. Potential numbers of regions are 4, 7, 10, 13, …

[0153] Fig. 9The possible partitioning of the potential representation after the 2D forward transform is illustrated. The potential representation is the sample point (potential sample point or quantized potential sample point) obtained after the 2D forward transform. The above potential samples are divided into 7 parts, which are represented as HH1, LH1, HL1, LL2, HL2, LH2 and HH2. HH1 describes that the part includes high-frequency components in the vertical direction and high-frequency components in the horizontal direction, and the partition depth is 1. HL2 describes that the part includes low-frequency components in the vertical direction, high-frequency components in the horizontal direction, and the partition depth is 2.

[0154] After the latent samples are obtained by the positive wavelet transform at the encoder, they are transmitted to the decoder using entropy coding and decoding. At the decoder, entropy decoding is applied to obtain the latent samples, which are then inversely transformed (by using Figure 7 The iWave inverse module in is used to obtain the reconstructed image.

[0155] 2.4 Neural Networks for Video Compression.

[0156] Similar to traditional video codec technology, neural image compression is the basis of intra-frame compression in neural network-based video compression. Therefore, the development of neural network-based video compression technology is later than that of neural network-based image compression, but due to its complexity, more efforts are needed to solve these challenges. Since 2017, some researchers have been studying neural network-based video compression schemes. Compared with image compression, video compression requires effective methods to eliminate redundancy between pictures. Inter-picture prediction is a key step in these works. Motion estimation and compensation are widely used, but have only recently been implemented by trained neural networks.

[0157] Neural network-based video compression research can be divided into two categories according to the target scenario: random access and low latency. In the case of random access, it requires that decoding can start from any point in the sequence, usually the entire sequence is divided into multiple separate segments, and each segment can be decoded independently. In the case of low latency, it aims to reduce the decoding time, so usually only the previous frame in the time domain can be used as a reference frame to decode the subsequent frame.

[0158] 2.4.1 Low latency.

[0159]

[29] was the first to propose a video compression scheme with a trained neural network. They first divided the video sequence frames into blocks, and each block would select one of the two available modes, intra-frame codec or inter-frame codec. When intra-frame codec is selected, an associated autoencoder is used to compress the block. When inter-frame codec is selected, motion estimation and compensation are performed using traditional methods, and residual compression is performed using a trained neural network. The output of the autoencoder is directly quantized and encoded using the Huffman method.

[0160] Chen et al.

[31] proposed another neural network-based video codec scheme with PixelMotionCNN. Frames are compressed sequentially in the temporal domain, and each frame is divided into blocks, which are compressed in raster scan order. Each frame will first be extrapolated with the previous two reconstructed frames. When a block is to be compressed, the extrapolated frame is fed into the PixelMotionCNN along with the context of the current block to derive a latent representation. The residual is then compressed through a variable rate image scheme

[34] . The performance of this scheme is comparable to that of H.264.

[0161] Lu et al.

[32] proposed a truly end-to-end neural network-based video compression framework, in which all modules are implemented using neural networks. The scheme accepts the current frame and the previous reconstructed frame as input, and uses a pre-trained neural network to derive optical flow as motion information. The motion information is warped together with the reference frame, and then the neural network generates a motion compensated frame. The residual and motion information are compressed using two separate neural autoencoders. The entire framework is trained using a single rate-distortion loss function. Its performance is better than H.264.

[0162] Rippel et al.

[33] proposed an advanced neural network-based video compression scheme. It inherits and extends the traditional video coding and decoding scheme with neural networks, with the following main features: 1) only one autoencoder is used to compress motion information and residuals; 2) motion compensation with multiple frames and multiple optical flows; 3) online states are learned and propagated through subsequent frames over time. The scheme has better performance than the HEVC reference software in terms of multi-scale structural similarity (MS-SSIM).

[0163] J.Lin et al.

[36] proposed an extended end-to-end neural network-based video compression framework based on

[32] . In this solution, multiple frames are used as references. Therefore, by using multiple reference frames and associated motion information, a more accurate prediction of the current frame can be provided. In addition, motion field prediction is deployed to eliminate motion redundancy along the temporal channel. A post-processing network is also introduced in this work to eliminate reconstruction artifacts in the previous process. Its performance is significantly better than

[32] and H.265 in terms of peak signal-to-noise ratio (PSNR) and MS-SSIM.

[0164] Eirikur et al.

[37] proposed scale-space flow to replace the commonly used optical flow by adding a scale parameter based on the framework of

[32] . Its performance is reported to be better than H.264.

[0165] Z.Hu et al.

[38] proposed a multi-resolution representation of optical flow based on

[32] . Specifically, the motion estimation network generates multiple optical flows with different resolutions and lets the network learn which one to choose under a loss function. The performance is slightly improved compared to

[32] and is better than H.265.

[0166] 2.4.2 Random access.

[0167] Wu et al.

[30] proposed a neural network-based video compression scheme with frame interpolation. The key frame is first compressed using a neural image compressor, and the remaining frames are compressed in a hierarchical order. They perform motion compensation in the perceptual domain, i.e., derive feature maps at multiple spatial scales of the original frame and use the motion to warp the feature maps, which are then used in the image compressor. The method is reported to be comparable to H.264.

[0168] Djelouah et al.

[41] proposed an interpolation-based video compression method in which the interpolation model combined motion information compression and image synthesis, and the same autoencoder was used for the image and the residual.

[0169] Amirhossein et al.

[35] proposed a neural network-based video compression method based on variational autoencoders and deterministic encoders. Specifically, the model includes an autoencoder and an autoregressive prior. Unlike previous methods, this method receives a group of pictures (GOP) as input and considers temporal correlation when encoding and decoding the latent representation, thereby introducing a 3D autoregressive prior. It provides performance comparable to H.265.

[0170] 2.5 Preparation.

[0171] Almost all natural images / videos are in digital format. Grayscale digital images can be obtained by Indicates that is the set of pixel values, m is the image height, and n is the image width. For example, is a common setting, and in this case Therefore, a pixel can be represented by an 8-bit integer. An uncompressed grayscale digital image has 8 bits per pixel (bpp), and compressed bits are certainly less.

[0172] Color images are usually represented by multiple channels to record color information. For example, in the RGB color space, an image can be represented by Representation, three separate channels store red, green and blue information. Similar to 8-bit grayscale images, uncompressed 8-bit RGB images have 24bpp. Digital images / videos can be represented in different color spaces. Neural network-based video compression schemes are mostly developed in RGB color space, while traditional codecs usually use YUV color space to represent video sequences. In YUV color space, the image is decomposed into three channels, namely Y, Cb and Cr, where Y is the brightness component and Cb / Cr are the chrominance components. The benefit comes from the fact that Cb and Cr are usually downsampled to achieve pre-compression because the human visual system is not very sensitive to the chrominance components.

[0173] A color video sequence consists of multiple color images called frames to record scenes at different time stamps. For example, in the RGB color space, a color video can be represented by X = {x0, x1, …, x t ,…,x T-1}, where T is the number of frames in the video sequence, If m=1080, n=1920, And the video has 50 frames per second (fps), then the data rate of this uncompressed video is 1920×1080×8×3×50=2,488,320,000 bits per second (bps), which is about 2.32 Gbps, which requires a lot of storage space, so it definitely needs to be compressed before transmitting over the Internet.

[0174] Typically lossless methods can achieve a compression ratio of about 1.5 to 3 for natural images, which is clearly lower than required. Therefore, lossy compression was developed to achieve higher compression ratios, but at the expense of distortion. Distortion can be measured by calculating the average squared difference between the original image and the reconstructed image, i.e., the mean square error (MSE). For grayscale images, MSE can be calculated using the following equation.

[0175]

[0176] Therefore, the quality of the reconstructed image compared to the original image can be measured by the Peak Signal-to-Noise Ratio (PSNR):

[0177]

[0178] in yes The maximum value of is, for example, 255 for an 8-bit grayscale image. There are other quality assessment metrics such as structural similarity (SSIM) and multi-scale SSIM (MS-SSIM) [4].

[0179] To compare different lossless compression schemes, one only needs to compare the compression ratio for a given rate, and vice versa. However, to compare different lossy compression methods, both the rate and the reconstruction quality must be considered. For example, it is common to calculate the relative rates for several different quality levels and then average the rates; the average relative rate is called the Bjortegaard delta rate (BD rate) [5]. There are other important aspects for evaluating image / video codecs, including encoding / decoding complexity, scalability, robustness, etc.

[0180] 3. The present disclosure.

[0181] The detailed techniques in this article should be considered as examples to explain the general concepts. These techniques should not be interpreted in a narrow way. In addition, these techniques can be combined in any way.

[0182] 3.1 Network architecture and processing steps.

[0183] 3.1.1 Decoder.

[0184] Figure 10A-10B An example of the structure of the proposed decoder architecture is illustrated. The decoding process consists of three different steps, which are performed one by one.

[0185] First, the entropy decoding process is performed and completed to obtain the quantized super potential value and the quantized residual potential value

[0186] Second, apply and complete the potential sample prediction process to obtain and Get quantified potential samples

[0187] Finally, a composite transformation process is applied to use Generate a reconstructed image.

[0188] 3.1.2. Entropy decoding process.

[0189] The entropy decoding process involves parsing two separate bitstreams that are packaged into a single file. The first bitstream ( Fig. 10A The discrete cumulative distribution function is stored in a predetermined fixed table and is used to resolve the quantized hyper-prior potential value Then, the quantized hyper-prior potential value is processed by the super prior variance decoder, which is a neural network-based subnetwork for generating the Gaussian variance σ. Then, assuming a zero-mean Gaussian distribution The quantized residual potential samples are obtained by applying arithmetic decoding to the second bitstream (bitstream2)

[0190] The modules involved in the entropy decoding process include entropy decoder, demasking and super prior variance decoder. It is worth noting that the entire entropy decoding process can be performed before the potential sample prediction process begins.

[0191] 3.1.3 Potential sample point prediction process.

[0192] At the beginning of the latent sample prediction process, the super prior variance decoder is used to decode the super prior latent sample. The inverse transform operation is performed. The output of this process is concatenated with the output of the context model module and then processed by the prediction fusion model to generate the prediction samples μ. The prediction samples are then added to the quantized residual samples To obtain the potential sample points for quantification

[0193] It is worth noting that the latent sample prediction process is an autoregressive process. However, due to the proposed architecture design, the quantized latent samples in different rows Can be processed in parallel.

[0194] The modules involved in the potential sample prediction process are marked in blue in Figure 11.

[0195] 3.1.4 Synthetic transformation process.

[0196] The synthesis transformation process is performed by the synthesis transformation module in Figure 11.

[0197] 3.2 Encoder.

[0198] The coding process includes analysis transformation, super analysis transformation, residual sample generation and entropy coding steps. Figure 11A-11B A structural example of the proposed encoder architecture is illustrated.

[0199] 3.2.1 Analyze the transformation process.

[0200] The analysis transform is the mirror image of the synthesis transform as described in Section 2.1.3. The input image is transformed into latent samples y using the analysis transform.

[0201] 3.2.2 Super analysis transformation process.

[0202] The super encoder module is a mirror operation of the super decoder as described in Section 2.1.2. The output of the super encoding process is rounded and included in the bitstream 1 via entropy coding and decoding.

[0203] 3.2.3 Residual sample generation process.

[0204] The residual sample generation process includes the latent sample prediction process as described in Section 2.1.2. After applying the sample prediction process, the predicted sample μ is obtained. The predicted sample is then subtracted from the latent sample y to obtain the residual sample, which is rounded to obtain the quantized residual sample

[0205] 3.2.4 Entropy coding process.

[0206] The entropy encoding process is a mirror image of the entropy decoding process as described in Section 2.1.1. The quantized residual samples are quantized using the Gaussian variance variable σ obtained as the output of the super-prior variance decoder. Perform entropy coding.

[0207] 3.3 Subnetwork.

[0208] Fig.12 Details of attention blocks, residual units, and residual blocks are illustrated.

[0209] Fig.13 The details of the residual down-sampling block and the residual up-sampling block are illustrated.

[0210] 3.4 Description of encoding and decoding tools.

[0211] 3.4.1 Decoupled entropy decoding and latent sample reconstruction (decoupled network).

[0212] The entropy decoding process uses arithmetic decoding, which is a completely sequential process with little possibility of parallelization. Although the bitstream can be divided into multiple sub-bitstreams to improve parallel processing capabilities, this will come at the cost of codec loss, and each bin in the sub-bitstream must be processed sequentially. Therefore, bitstream parsing is completely unsuitable for processing units capable of massive parallel processing (such as GPUs or NPUs), which is the ultimate goal of future end-to-end image codecs.

[0213] This problem has been recognized in the development of state-of-the-art video codec standards such as HEVC and VVC. In such standards, the parsing of the bitstream via the CABAC engine is performed completely independently of the sample reconstruction. This allows the development of a dedicated engine for CABAC that starts parsing the bitstream before starting sample reconstruction. Bitstream parsing is the absolute bottleneck of the decoding process chain, and the design principles followed by HEVC and VVC allow CABAC to be performed without waiting for any sample reconstruction process.

[0214] Although the above-mentioned parsing independence principle is strictly adhered to in HEVC and VVC, the decoding time of the latest E2E image codec architectures that can achieve competitive codec gains is very slow due to this problem. Architectures such as [3] use autoregressive processing units in the entropy codec unit, which makes them incompatible with massively parallel processing units and thus has extremely slow decoding time.

[0215] One of the core algorithms in our submission is a network architecture that is able to parse the bitstream independently of the latent sample reconstruction, referred to as the “decoupled network”. In the decoupled network, two super decoders are used instead of just one, called the super decoder and the super prior variance decoder. The super prior variance decoder generates the Gaussian variance parameters σ and is part of the entropy decoding process. On the other hand, the super decoder is part of the latent sample reconstruction process and participates in generating the latent prediction samples μ. During the entropy decoding process, only σ is used to quantize the residual samples. Therefore, the entropy decoding process can be performed completely independently from the sample reconstruction process.

[0216] Decoding the quantized residual samples from the bitstream Finally, using the fully available input and The latent sample reconstruction process is started. The modules involved in this process are the super decoder, context model and prediction fusion model, which are all neural network units that require a large number of calculations that can be performed in parallel. Therefore, the latent sample prediction process is now suitable for execution in a GPU-like processing unit, which provides a huge advantage in implementation flexibility and a huge improvement in decoding speed.

[0217] 3.4.2 Wavefront Parallel Processing (WPP).

[0218] Fig.14 The masked convolution kernel used is illustrated. In the submitted solution, Fig.14 The 2D masked convolution kernel depicted in .

[0219] In order to improve the utilization of GPU, a wavefront parallel processing mechanism is introduced in the potential sample prediction process. The core of the context model module is Fig.14 where the sample to be predicted is at 0,0 coordinates. The kernel is designed in such a way that a row of samples can be processed in parallel with the previous row of samples, delayed by only one sample. The sample processing mode is Fig.15 Depicted in.

[0220] 3.4.3 Color format conversion.

[0221] During the encoding / decoding process, the proposed scheme supports 8-bit and 16-bit input images, and the decoded images can be saved as 8-bit or 16-bit. During the training process, the input images are converted to YUV space according to the BT.709 specification [5]. The training metrics are calculated in the YUV color space using weighted losses of the luminance and chrominance components.

[0222] 3.4.4 Adaptive Quantization (AQ).

[0223] In the decoder, modules MASK&SCALE[1] and MASK&SCALE[2]. This operation includes the following steps:

[0224] 1. Determine the mask for each potential sample point using the following formula:

[0225]

[0226] 2. Based on the value of the mask, apply a scaling operation to the quantized residual samples and Gaussian variance samples:

[0227] MASK&SCALE[1]:

[0228] MASK&SCALE[2]:

[0229] In the encoder, additionally, the MASK&SCALE[5] module participates in adaptive quantization and performs the following operations:

[0230] MASK&SCALE[5]: where w[c,i,j] is the unquantized residual potential sample, and the "thr", "scale" and "greater_flag" parameters are signaled in the bitstream as part of the adaptive mask and scaling syntax table (Section 4.1). All three processing modules MASK&SCALE[1], MASK&SCALE[2] and MASK&SCALE[5] use the same mask.

[0231] The adaptive quantization process can be performed multiple times one by one to modify and σ. The number of operations is signaled in the bitstream by num_adaptive_quant_param (Section 4.1). By default, the value of this parameter is set to 3, and the precomputed "thr", "scale", and "greater_flag" values ​​are signaled in the bitstream for each process.

[0232] The adaptive quantization process controls the quantization step size of each residual potential sample according to the estimated variance σ.

[0233] 3.4.5 Latent Scaling Before Synthesis (LSBS).

[0234] In the decoder, the modules MASK&SCALE[3] and MASK&SCALE[4] participate in this process and are only applied at the decoder. The operation consists of the following steps:

[0235] 1. Determine the mask for each potential sample point using the following formula:

[0236]

[0237] 2. Based on the value of the mask, the value of the reconstructed potential sample is modified as follows:

[0238]

[0239] where "thr", "scale", "scale2" and "greater_flag" and parameters are signaled in the bitstream as part of the Adaptive Mask and Scaling Syntax Table (Section 4.1).

[0240] By default, 2 sets of LSBS parameter sets are signaled in the bitstream and are applied one by one according to the order in which they are signaled. The number of LSBS parameters is controlled by the num_latent_post_process_params syntax element.

[0241] The same syntax table is used for signaling of adaptive quantization and potential scaling before synthesis (Section 4.1), and the two processing modes are identified by the "mode" parameter. When the mode parameter is set to 5, an additional "scale2" parameter is signaled before synthesis to obtain the potential scaling.

[0242] 3.4.6 Latent Domain Adaptive Offset (LDAO).

[0243] This process is applied before the synthesis transform process. First, for each of the 192 channels of the latent code, a flag is signaled to indicate whether an offset is present (such as the offsets_signalled[idx] flag in Section 4.2). In addition, the latent code is divided into slices horizontally and vertically (the number of vertical and horizontal partitions are represented by the num_horizontal_split and num_vertical_split variables). Finally, for each channel of each slice, if offsets_signalled[idx] is true for that channel, the offset value is signaled for that channel. The offset value is signaled using a fixed length codec (8 bits) and is in an absolute manner without a predictive codec.

[0244] The LDAO tool helps to offset the quantization noise introduced on the potential samples of the quantization. The encoder calculates the offset value so that minimize.

[0245] 3.4.7 Block-based residual skipping.

[0246] A block-based residual skip mode is designed, where the residual is optionally encoded into the bitstream, i.e., some residuals are skipped and not encoded into the bitstream. The residual map is divided into blocks. Depending on the statistics of the residual block, if the percentage of zero entries is greater than a predefined threshold, the residual block is skipped. This indicates that the residual contains less information and skipping these residual blocks can achieve a better complexity-performance trade-off.

[0247] 3.4.8 Reconstruction and resampling.

[0248] Reconstruction resampling allows for flexible selection of models from a model set while still achieving the target rate. In E2E-based image codecs, very annoying color artifacts may be introduced in the reconstructed image of low-rate data points. In our solution, if such a problem is found, the input image is downsampled and the downsampled image is encoded using a model designed for a higher rate. This effectively solves the color shift problem, but at the expense of sample fidelity.

[0249] 3.4.9 Quantization of the Super-Prior Variance Decoder Network.

[0250] In arithmetic coding and decoding, quantized latent features are encoded into the bitstream according to the probabilities obtained from the entropy model and reconstructed from the bitstream through the inverse operation. In this case, even a small change in the probability modeling will lead to a completely different result, because such a small difference can be propagated in the subsequent process and eventually introduce the problem of undecodability. To alleviate this problem and achieve device interoperability in practical image compression, a neural network quantization strategy is proposed. Due to the decoupled entropy module, we only need scale information when performing arithmetic coding and decoding, which means that we only need to quantize the super-scale decoding network to ensure multi-device consistency of scale information reasoning. Two parameters, the scaling parameter and the upper limit parameter, are stored together with the model weights. The scaling parameter scales the weights of the network and input values ​​to a fixed precision and avoids numerical overflow of the neural network calculation, which is the main factor affecting device interoperability. In our solution, the quantized weights and values ​​are set to 16 bits, the scaling parameter is always a power of 2, and the detailed value depends on the potential maximum value of the weights and inputs we observed in step 1. To further avoid overflow of calculations in the intermediate network layers, the upper limit parameter is introduced to limit the value of the output of each layer.

[0251] It should be noted that the quantization and quantization calculation of the network are only performed after the model is trained. During the training phase, floating point precision is still used to estimate the rate and implement the back propagation of the neural network.

[0252] 3.4.10 Compositing Transformed Tiles.

[0253] The spatial size of the feature map increases significantly after being fed through the synthesis transformation network. During decoding, the problem of insufficient memory arises when the image is too large or the decoder has a limited memory budget. To address this issue, we designed a tiled segmentation for the synthesis neural network, which usually requires the largest memory budget during decoding. As illustrated in the figure below, the feature map is spatially segmented into multiple parts. Each segmented feature map will be fed through the subsequent convolutional layers one by one. After the most computationally intensive process is completed, they are cropped and stitched together to restore the spatial size. The segmentation type can be vertical, horizontal, or any combination of both, depending on the image size and memory budget. To mitigate potential reconstruction artifacts due to boundary effects (usually caused by padding), each sub-part has an associated padding region. The padding region is usually filled with neighboring values ​​in the feature map. Fig.16 Examples of vertical and horizontal tiling of the synthesis transformation network are illustrated.

[0254] 3.4.11 Entropy decoding.

[0255] Entropy decoding converts the bitstream into quantized features according to the probability table obtained from the entropy encoding and decoding model. In our solution, we only need to The Gaussian variance parameter σ obtained from the samples is used to decode the bit stream and generate quantized residual potential samples An asymmetric number system is used for symbol decoding.

[0256] 3.5 High-level syntax.

[0257]

[0258] 3.5.1 Adaptive Masking and Scaling Syntax.

[0259] The syntax table described below includes parameters used in performing latent scaling before synthesis (LSBS), adaptive quantization (AQ), and block-based residual skipping.

[0260]

[0261]

[0262] 3.5.2 Adaptive Offset Syntax.

[0263] The syntax table described below includes parameters used in performing Latent Domain Adaptive Offset (LDAO).

[0264]

[0265]

[0266] 3.6 Encoder algorithm.

[0267] All encoder configuration parameters required by the encoder are pre-optimized. The encoder's prepare_weights() function calculates the default pre-optimized encoder configuration parameters, and the write_weights() function includes them in the high-level syntax part of the bitstream.

[0268] Since RDO is not performed, the decoding process is not performed as part of the encoding, and the encoding process is as fast as the decoding process. The encoding time is about 1.6 times the decoding time using GPU processing.

[0269] Some encoder configuration parameters required to encode the image in the submitted file are slightly different from the default pre-optimized configuration parameters. This is because some parameters of some images and rate points are manually modified during the rate matching process (such as those belonging to adaptive quantization). In rare cases, manual parameter adjustments are also applied to fix visual artifacts. If rate matching does not need to be applied, the encoding process does not require any specific strategy and the encoder uses the predefined default encoder configuration parameters.

[0270] 3.6.1 Online Potential Optimization Before Quantization.

[0271] In the encoder, after applying the analysis transform and obtaining the unquantized latent samples y, an online refinement iteration is applied. ) is inversely transformed and the reconstructed image is used to calculate the MSE loss. One backpropagation iteration is performed using the MSE loss to refine the sample y. In other words, the online latent optimization consists of only one forward and one backpropagation process.

[0272] The online latent refinement is intentionally kept simple in order not to increase the encoding time. Furthermore, although increasing the number of iterations can increase the gain, only one iteration is applied to limit the increase in encoding time.

[0273] 3.7 Training instructions.

[0274] 3.7.1 Objective Submission.

[0275] Training data. The training set recommended by JPEG-AI was used throughout the training process. When preparing the training data, the original images were resized to multiple sizes and randomly cropped into small training blocks.

[0276] Training details. We trained 16 models with different Lagrange multipliers. The training process was multi-stage. In the first stage, we trained 5 models with 200 epochs. In the first training stage, the super-prior variance decoder module including another 3 additional layers was used. In the second stage, the longer super-prior variance decoder network was replaced with the network depicted in Figure 11, and the entire encoder-decoder was trained for 100 epochs. Finally, in the third stage, we used the 5 trained models from the second stage to obtain the other 11 models by fine-tuning. The initial learning rates of the three stages were set to 1e-4, 1e-5, and 1e-5, respectively. The Adam optimizer and the plateau-reducing learning rate scheduler were used throughout the training process.

[0277] 3.7.2 Subjective Submission.

[0278] To further improve the visual quality, we analyze the relationship between rate, objective solution, and perception-based solution. For the low-rate model, we utilize five additionally trained perception-based models to improve the subjective quality of the corresponding rate points. Specifically, to obtain these perception-based models, we use the corresponding objective-oriented models as starting points and train five models at a lower rate using a perceptual loss function. The perceptual loss is defined as follows:

[0279]

[0280] Among them G loss is the loss of the discriminator, LPIPS is learned to perceive image patch similarity [4], and the setting of λ follows the setting of the target-oriented model.

[0281] In the discriminator, we use As conditional inputs, the original YUV and reconstructed YUV will be fed into the discriminator separately to test whether the input is real (original image) or fake (distorted image). Through the training process, our perception-based model will learn to be as close to the original image as possible in visual quality. The structure of the discriminator is shown in Fig.17 Shown in.

[0282] It is worth noting that the discriminator is only used during the training process and is not included in the final model.

[0283] For more details on references, see:

[0284] [1]Z.Cheng,H.Sun,M.Takeuchi and J.Katto,“Learned image compressionwith discretized gaussian mixture likelihoods and attention modules,”inProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition,pp.7939-7948,2020.

[0285] [2]B.Bross,J.Chen,S.Liu and Y.-K.Wang,“Versatile Video Draft(Draft10),”JVET-S2001,Jul.2020.

[0286] [3]R.D.Dony and S.Haykin,“Neural network approaches to imagecompression,”Proceedings of the IEEE,vol.83,no.2,pp.288–303,1995.

[0287] [4]Z.Wang,A.C.Bovik,H.R.Sheikh,and E.P.Simoncelli,“Image qualityassessment:From error visibility to structural similarity,”IEEE Transactionson Image Processing,vol.13,no.4,pp.600–612,2004.

[0288] [5]G.Bjontegaard,“Calculation of average PSNR differences between RD-curves,”VCEG,Tech.Rep.VCEG-M33,2001.

[0289] [6]C.E.Shannon,“Amathematical theory of communication,”Bell SystemTechnical Journal,vol.27,no.3,pp.379–423,1948.

[0290]

[17] G.E.Hinton and R.R.Salakhutdinov,“Reducing the dimensionality ofdata with neural networks,”Science,vol.313,no.5786,pp.504–507,2006.

[0291]

[18] G.Toderici,S.M.O’Malley,S.J.Hwang,D.Vincent,D.Minnen,S.Baluja,M.Covell,and R.Sukthankar,“Variable rate image compression with recurrentneural networks,”arXiv preprint arXiv:1511.06085,2015.

[0292]

[19] G.Toderici,D.Vincent,N.Johnston,S.J.Hwang,D.Minnen,J.Shor,andM.Covell,“Full resolution image compression with recurrent neural networks,”in CVPR,2017,pp.5306–5314.

[0293]

[20] N.Johnston,D.Vincent,D.Minnen,M.Covell,S.Singh,T.Chinen,S.JinHwang,J.Shor,and G.Toderici,“Improved lossy image compression with primingand spatially adaptive bit rates for recurrent networks,”in CVPR,2018,pp.4385–4393.

[0294]

[21] M.Covell,N.Johnston,D.Minnen,S.J.Hwang,J.Shor,S.Singh,D.Vincent,and G.Toderici,“Target-quality image compression with recurrent,convolutionalneural networks,”arXiv preprint arXiv:1705.06687,2017.

[0295]

[22] J.Ballé,V.Laparra,and E.P.Simoncelli,“End-to-end optimization ofnonlinear transform codes for perceptual quality,”in PCS.IEEE,2016,pp.1–5.

[0296]

[23] J.Ballé,“Efficient nonlinear transforms for lossy imagecompression,”in′PCS,2018,pp.248–252.

[0297]

[24] J.Ballé,V.Laparra and E.P.Simoncelli,“End-to-end optimized imagecompression,”in International Conference on Learning Representations,2017.

[0298]

[25] J.Ballé,D.Minnen,S.Singh,S.Hwang and N.Johnston,“Variationalimage compression with a scale hyperprior,”in International Conference onLearning Representations,2018.

[0299]

[26] D.Minnen,J.Ballé,G.Toderici,“Joint Autoregressive andHierarchical Priors for Learned Image Compression”,arXiv.1809.02736.1,2,3,4,7

[0300]

[27] Z.Cheng,H.Sun M.Takeuchi and J.Katto,“Learned image compressionwith discretized Gaussian mixture likelihoods and attention modules,”inProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition(CVPR),2020.

[0301]

[28] Github repository“CompressAI:https: / / github.com / InterDigitalInc / CompressAI,”,InterDigital Inc,accessed Dec2020.

[0302]

[29] T.Chen,H.Liu,Q.Shen,T.Yue,X.Cao,and Z.Ma,“DeepCoder:Adeep neuralnetwork based video compression,”in VCIP.IEEE,2017,pp.1–4.

[0303]

[30] C.-Y.Wu,N.Singhal,and P.Krahenbuhl,“Video compression throughimage interpolation,”in Proceedings of the European Conference on ComputerVision(ECCV),2018,pp.416–431.

[0304]

[31] Z.Chen,T.He,X.Jin,and F.Wu,“Learning for video compression,”IEEETransactions on Circuits and Systems for Video Technology,DOI:10.1109 / TCSVT.2019.2892608,2019.

[0305]

[32] G.Lu,W.Ouyang,D.Xu,X.Zhang,C.Cai,and Z.Gao,“DVC:An end-to-enddeep video compression framework,”in Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition(CVPR),2019.

[0306]

[33] O.Rippel,S.Nair,C.Lew,S.Branson,A.Anderson and L.Bourdev,"LearnedVideo Compression,"2019 IEEE / CVF International Conference on Computer Vision(ICCV),Seoul,Korea(South),2019,pp.3453-3462,doi:10.1109 / ICCV.2019.00355.

[0307]

[34] G.Toderici,D.Vincent,N.Johnston,S.J.Hwang,D.Minnen,J.Shor,andM.Covell,“Full resolution image compression with recurrent neural networks,”in CVPR,2017,pp.5306–5314.

[0308]

[35] A.Habibian,T.Rozendaal,J.Tomczak and T.Cohen,“Video Compressionwith Rate-Distortion Autoencoders,”in Proceedings of the IEEE / CVFInternational Conference on Computer Vision(ICCV),2019,pp.7033-7042.

[0309]

[36] J.Lin,D.Liu,H.Li and F.Wu,“M-LVC:Multiple frames prediction forlearned video compression,”in Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition(CVPR),2020.

[0310]

[37] E.Agustsson,D.Minnen,N.Johnston,J.Ballé,S.J.Hwang andG.Toderici,"Scale-Space Flow for End-to-End Optimized Video Compression,"2020IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR),Seattle,WA,USA,2020,pp.8500-8509,doi:10.1109 / CVPR42600.2020.00853.

[0311]

[38] X.Hu,Z.Chen,D.Xu,G.Lu,W.Ouyang and S.Gu,“Improving deep videocompression by resolution-adaptive flow coding,”in European Conference onComputer Vision(ECCV)2020.

[0312]

[39] B.Li,H.Li,L.Li and J.Zhang,"λDomain Rate Control Algorithm forHigh Efficiency Video Coding,"in IEEE Transactions on Image Processing,vol.23,no.9,pp.3841-3854,Sept.2014,doi:10.1109 / TIP.2014.2336550.

[0313]

[40] L.Li,B.Li,H.Li and C.W.Chen,"λDomain Optimal Bit AllocationAlgorithm for High Efficiency Video Coding,"in IEEE Transactions on Circuitsand Systems for Video Technology,vol.28,no.1,pp.130-142,Jan.2018,doi:10.1109 / TCSVT.2016.2598672.

[0314]

[41] Abdelaziz Djelouah,Joaquim Campos,Simone Schaub-Meyer,andChristopher Schroers.Neural inter-frame com-pression for video coding.InICCV,pages 6421–6429,October 2019.

[0315]

[42] F.Bossen,Common Test Conditions and Software ReferenceConfigurations,document Rec.JCTVC-J1100,Stockholm,Sweden,Jul.2012.

[0316] Fig.184000 is a block diagram illustrating an example video processing system 4000 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of system 4000. System 4000 may include an input 4002 for receiving video content. The video content may be received in a raw or uncompressed format, such as 8-bit or 10-bit multi-component pixel values, or may be received in a compressed or encoded format. Input 4002 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces, such as Ethernet, a passive optical network (PON), and wireless interfaces, such as Wi-Fi or a cellular interface.

[0317] System 4000 may include a codec component 4004 that can implement various codecs or encoding methods described in this document. Codec component 4004 can reduce the average bit rate of the video from input 4002 to the output of codec component 4004 to generate a codec representation of the video. Therefore, codec technology is sometimes referred to as video compression or video transcoding technology. The output of codec component 4004 can be stored or transmitted via a connected communication (as represented by component 4006). The bitstream (or codec) representation of the storage or transmission of the video received at input 4002 can be used by component 4008 to generate pixel values ​​or displayable videos sent to display interface 4010. The process of generating a user-viewable video from a bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "codec" operations or tools, it should be understood that codec tools or operations are used at the encoder, and the corresponding decoding tools or operations of the reverse codec results will be performed by the decoder.

[0318] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB) or High Definition Multimedia Interface (HDMI) or Display Port, etc. Examples of storage interfaces include SATA (Serial Advanced Technology Attachment), PCI, IDE interfaces, etc. The technology described in this document may be embodied in various electronic devices, such as mobile phones, laptops, smart phones, or other devices capable of performing digital data processing and / or video display.

[0319] Fig.194106 is a block diagram of an example video processing device 4100. Device 4100 can be used to implement one or more methods described herein. Device 4100 can be embodied in a smartphone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. Device 4100 may include one or more processors 4102, one or more memories 4104, and video processing circuitry 4106. (Multiple) processors 4102 can be configured to implement one or more methods described in this document. (Multiple) memories 4104 can be used to store data and code for implementing the methods and techniques described herein. Video processing circuitry 4106 can be used to implement some of the techniques described in this document in hardware circuits. In some embodiments, video processing circuitry 4106 can be at least partially included in processor 4102, such as a graphics coprocessor.

[0320] Fig. 20 4200. The method 4200 includes determining to apply a pre-processing function to visual media data as part of an image compression framework at step 4202. At step 4204, performing a conversion between the visual media data and a bitstream based on the image compression framework. The conversion of step 4204 may include encoding at an encoder or decoding at a decoder, depending on the example.

[0321] It should be noted that the method 4200 may be implemented in an apparatus for processing video data, the apparatus comprising a processor and a non-transitory memory having instructions thereon, such as the video encoder 4400, the video decoder 4500, and / or the encoder 4600. In such a case, once the instructions are executed by the processor, the processor is caused to perform the method 4200. In addition, the method 4200 may be performed by a non-transitory computer-readable medium, the medium comprising a computer program product for use by the video codec apparatus. The computer program product comprises computer executable instructions stored on a non-transitory computer-readable medium, such that when the computer executable instructions are executed by the processor, the video codec apparatus is caused to perform the method 4200.

[0322] Fig.21 43 is a block diagram illustrating an example video codec system 4300 that can utilize the techniques of the present disclosure. The video codec system 4300 can include a source device 4310 and a target device 4320. The source device 4310 generates encoded video data and can be referred to as a video encoding device. The target device 4320 can decode the encoded video data generated by the source device 4310 and can be referred to as a video decoding device.

[0323] Source device 4310 may include a video source 4312, a video encoder 4314, and an input / output (I / O) interface 4316. Video source 4312 may include a source such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of these sources. Video data may include one or more pictures. Video encoder 4314 encodes video data from video source 4312 to generate a bitstream. A bitstream may include a sequence of bits that form a coded representation of video data. A bitstream may include coded pictures and associated data. A coded picture is a coded representation of a picture. Associated data may include sequence parameter sets, picture parameter sets, and other syntax structures. I / O interface 4316 may include a modulator / demodulator (modem) and / or a transmitter. The coded video data may be directly transmitted to target device 4320 via network 4330 via I / O interface 4316. The coded video data may also be stored on storage medium / server 4340 for access by target device 4320.

[0324] Target device 4320 may include an I / O interface 4326, a video decoder 4324, and a display device 4322. The I / O interface 4326 may include a receiver and / or a modem. The I / O interface 4326 may obtain encoded video data from source device 4310 or storage medium / server 4340. The video decoder 4324 may decode the encoded video data. The display device 4322 may display the decoded video data to a user. The display device 4322 may be integrated with the target device 4320, or may be external to the target device 4320, and may be configured to interface with an external display device.

[0325] The video encoder 4314 and the video decoder 4324 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVM) standard, and other current and / or later standards.

[0326] Fig. 22 is a block diagram illustrating an example of a video encoder 4400, which may be Fig.21 4300. Video encoder 4400 may be configured to perform any or all of the techniques of the present disclosure. Video encoder 4400 includes multiple functional components. The techniques described in the present disclosure may be shared between the various components of video encoder 4400. In some examples, the processor may be configured to perform any or all of the techniques described in the present disclosure.

[0327] The functional components of the video encoder 4400 may include a segmentation unit 4401, a prediction unit 4402 (which may include a mode selection unit 4403), a motion estimation unit 4404, a motion compensation unit 4405, an intra-frame prediction unit 4406, a residual generation unit 4407, a transform processing unit 4408, a quantization unit 4409, an inverse quantization unit 4410, an inverse transform unit 4411, a reconstruction unit 4412, a buffer 4413 and an entropy coding unit 4414.

[0328] In other examples, the video encoder 4400 may include more, fewer, or different functional components. In one example, the prediction unit 4402 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode in which at least one reference picture is a picture in which the current video block is located.

[0329] Furthermore, some components, such as the motion estimation unit 4404 and the motion compensation unit 4405 , may be highly integrated but are represented separately in the example of the video encoder 4400 for purposes of explanation.

[0330] The segmentation unit 4401 may segment the picture into one or more video blocks. The video encoder 4400 and the video decoder 4500 may support various video block sizes.

[0331] The mode selection unit 4403 may select one of the coding modes (intra or inter) based on the error result, for example, and provide the resulting intra or inter coded block to the residual generation unit 4407 to generate residual block data, and to the reconstruction unit 4412 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 4403 may select a combination of intra and inter prediction (CIIP) modes, where the prediction is based on an inter prediction signal and an intra prediction signal. In the case of inter prediction, the mode selection unit 4403 may also select a resolution (e.g., sub-pixel or integer pixel precision) for the motion vector of the block.

[0332] In order to perform inter-frame prediction on the current video block, the motion estimation unit 4404 may generate motion information of the current video block by comparing the one or more reference frames from the buffer 4413 with the current video block. The motion compensation unit 4405 may determine a predicted video block of the current video block based on the motion information and decoded samples of pictures from the buffer 4413 other than the picture associated with the current video block.

[0333] The motion estimation unit 4404 and the motion compensation unit 4405 may perform different operations on the current video block, for example, depending on whether the current video block is in an I slice, a P slice, or a B slice.

[0334] In some examples, the motion estimation unit 4404 may perform unidirectional prediction on the current video block, and the motion estimation unit 4404 may search for a reference video block of the current video block in the reference picture of list 0 or list 1. The motion estimation unit 4404 may then generate a reference index indicating a reference picture containing the reference video block in list 0 or list 1, and a motion vector indicating a spatial displacement between the current video block and the reference video block. The motion estimation unit 4404 may output the reference index, the prediction direction indicator, and the motion vector as motion information of the current video block. The motion compensation unit 4405 may generate a predicted video block of the current block based on the reference video block indicated by the motion information of the current video block.

[0335] In other examples, the motion estimation unit 4404 may perform bidirectional prediction on the current video block, and the motion estimation unit 4404 may search for a reference video block of the current video block in the reference pictures in list 0, and may also search for another reference video block of the current video block in the reference pictures in list 1. The motion estimation unit 4404 may then generate a reference index and a motion vector, the reference index indicating the reference pictures containing the reference video block in list 0 and list 1, and the motion vector indicating the spatial displacement between the reference video block and the current video block. The motion estimation unit 4404 may output the reference index and the motion vector of the current video block as the motion information of the current video block. The motion compensation unit 4405 may generate a predicted video block of the current video block based on the reference video block indicated by the motion information of the current video block.

[0336] In some examples, motion estimation unit 4404 may output a full set of motion information for a decoding process of a decoder. In some examples, motion estimation unit 4404 may not output a full set of motion information for the current video. Instead, motion estimation unit 4404 may signal the motion information of the current video block with reference to the motion information of another video block. For example, motion estimation unit 4404 may determine that the motion information of the current video block is sufficiently similar to the motion information of an adjacent video block.

[0337] In one example, the motion estimation unit 4404 may indicate a value in a syntax structure associated with the current video block that indicates to the video decoder 4500 that the current video block has the same motion information as another video block.

[0338] In another example, the motion estimation unit 4404 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 4500 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0339] As discussed above, the video encoder 4400 may predictively signal motion vectors.Two examples of predictive signaling techniques that may be implemented by the video encoder 4400 include Advanced Motion Vector Prediction (AMVP) and merge mode signaling.

[0340] The intra prediction unit 4406 may perform intra prediction on the current video block. When the intra prediction unit 4406 performs intra prediction on the current video block, the intra prediction unit 4406 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include the predicted video block and various syntax elements.

[0341] The residual generation unit 4407 may generate residual data for the current video block by subtracting the prediction video block(s) of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample components of the samples in the current video block.

[0342] In other examples, the current video block may not have residual data for the current video block, for example, in skip mode, the residual generation unit 4407 may not perform a subtraction operation.

[0343] The transform processing unit 4408 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0344] After the transform processing unit 4408 generates a transform coefficient video block associated with the current video block, the quantization unit 4409 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0345] The inverse quantization unit 4410 and the inverse transform unit 4411 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 4412 may add the reconstructed residual video block to the corresponding samples of one or more prediction video blocks generated by the prediction unit 4402 to generate a reconstructed video block associated with the current block for storage in the buffer 4413.

[0346] After the reconstruction unit 4412 reconstructs the video block, a loop filtering operation may be performed to reduce video block artifacts in the video block.

[0347] The entropy coding unit 4414 may receive data from other functional components of the video encoder 4400. When the entropy coding unit 4414 receives the data, the entropy coding unit 4424 may perform one or more entropy coding operations to generate entropy coded data and output a bitstream including the entropy coded data.

[0348] Fig.23 is a block diagram illustrating an example of a video decoder 4500, which may be Fig.21 4300 in the system 4300 illustrated in FIG. 4324. Video decoder 4500 may be configured to perform any or all of the techniques of the present disclosure. In the example shown, video decoder 4500 includes multiple functional components. The techniques described in the present disclosure may be shared between the various components of video decoder 4500. In some examples, the processor may be configured to perform any or all of the techniques described in the present disclosure.

[0349] In the example shown, the video decoder 4500 includes an entropy decoding unit 4501, a motion compensation unit 4502, an intra prediction unit 4503, an inverse quantization unit 4504, an inverse transform unit 4505, a reconstruction unit 4506, and a buffer 4507. In some examples, the video decoder 4500 may perform a decoding process that is generally reciprocal to the encoding process described with respect to the video encoder 4400.

[0350] The entropy decoding unit 4501 may retrieve a coded bitstream. The coded bitstream may include entropy coded video data (e.g., coded blocks of video data). The entropy decoding unit 4501 may decode the entropy coded video data, and the motion compensation unit 4502 may determine motion information from the entropy decoded video data, including motion vectors, motion vector precision, reference picture list indexes, and other motion information. The motion compensation unit 4502 may determine such information, for example, by performing AMVP and merge mode.

[0351] The motion compensation unit 4502 may generate a motion compensated block, possibly performing interpolation based on an interpolation filter. An identifier of an interpolation filter for use with sub-pixel precision may be included in a syntax element.

[0352] The motion compensation unit 4502 may calculate interpolation of sub-integer pixels of a reference block using an interpolation filter used by the video encoder 4400 during encoding of the video block. The motion compensation unit 4502 may determine the interpolation filter used by the video encoder 4400 according to received syntax information and use the interpolation filter to generate a prediction block.

[0353] The motion compensation unit 4502 can use some syntax information to determine the size of the blocks used to encode (multiple) frames and / or (multiple) slices of the encoded video sequence, partitioning information describing how each macroblock of the encoded video sequence picture is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-frame codec block, and other information for decoding the encoded video sequence.

[0354] The intra prediction unit 4503 may form a prediction block from spatially adjacent blocks using, for example, an intra prediction mode received in the bitstream. The inverse quantization unit 4504 inversely quantizes, i.e., dequantizes, the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 4501. The inverse transform unit 4505 applies an inverse transform.

[0355] The reconstruction unit 4506 may add the residual block to the corresponding prediction block generated by the motion compensation unit 4502 or the intra prediction unit 4503 to form a decoded block. If necessary, a deblocking filter may also be applied to filter the decoded block to remove blocky artifacts. The decoded video block is then stored in a buffer 4507, which provides reference blocks for subsequent motion compensation / intra prediction and also generates a decoded video for presentation on a display device.

[0356] Fig.24 4600 is a schematic diagram of an example encoder 4600. The encoder 4600 is suitable for implementing VVC technology. The encoder 4600 includes three loop filters, namely a deblocking filter (DF) 4602, a sample adaptive offset (SAO) 4604, and an adaptive loop filter (ALF) 4606. Unlike the DF 4602 that uses a predefined filter, the SAO 4604 and the ALF 4606 use the original samples of the current image to reduce the mean square error between the original samples and the reconstructed samples by adding an offset and applying a finite impulse response (FIR) filter, respectively, where the offset and filter coefficients are signaled by coded auxiliary information. The ALF 4606 is located at the last processing stage of each picture and can be regarded as a tool that attempts to capture and repair artifacts produced by the previous stage.

[0357] The encoder 4600 also includes an intra prediction component 4608 and a motion estimation / compensation (ME / MC) component 4610 configured to receive an input video. The intra prediction component 4608 is configured to perform intra prediction, while the ME / MC component 4610 is configured to perform inter prediction using a reference picture obtained from a reference picture buffer 4612. The residual block from the inter prediction or intra prediction is fed to a transform (T) component 4614 and a quantization (Q) component 4616 to generate quantized residual transform coefficients, which are fed to an entropy codec component 4618. The entropy codec component 4618 entropy encodes the prediction result and the quantized transform coefficients and sends them to a video decoder (not shown). The quantization component output from the quantization component 4616 can be fed to an inverse quantization (IQ) component 4620, an inverse transform component 4622, and a reconstruction (REC) component 4624. The REC component 4624 is capable of outputting images to the DF 4602 , SAO 4604 , and ALF 4606 for filtering before storing the images in the reference picture buffer 4612 .

[0358] Fig.25 The method 2500 for decoding an image according to an embodiment of the present disclosure may be implemented by a decoding device (eg, a decoder). In block 2502, the decoding device performs an entropy decoding process to obtain quantized super potential samples. and quantized residual potential samples

[0359] In block 2504, the encoding device applies a potential sample prediction process to obtain a quantized super potential sample. and quantized residual potential samples Get quantified potential samples In block 2506, the encoding device applies a synthetic transform process to use the quantized potential samples Generate a reconstructed image.

[0360] Fig.26 The image encoding method 2600 according to an embodiment of the present disclosure may be implemented by an encoding device (eg, an encoder). In block 2602, the encoding device transforms an input image into potential samples y using an analysis transform.

[0361] In block 2604, the encoding device quantizes the potential sample y using a super encoder to generate a quantized super potential sample In block 2606, the encoding device entropy codes the quantized super potential samples In block 2608, the encoding device applies a latent sample prediction process to use the quantized super latent samples Get quantized potential samples based on potential samples y and quantized residual potential samples

[0362] In block 2610, the encoding device obtains the predicted sample μ after the potential sample prediction process. In block 2612, the encoding device converts the quantized super potential sample μ into and quantized residual samples Entropy encoding into a bit stream.

[0363] A list of some example preferred solutions is provided below.

[0364] The following solutions illustrate examples of the techniques discussed herein.

[0365] 1. An image decoding method comprising the steps of: using an arithmetic decoder to obtain a reconstructed potential value The reconstructed potential value is fed into the synthetic neural network; based on the decoding parameters for tile segmentation, the output feature map is tiled into multiple parts at one or more locations; each part is fed into the next level of convolutional layer to obtain the output spatially segmented feature map; the spatially segmented feature map is cropped and spliced ​​into a spatially complete feature map; continue until the image is reconstructed.

[0366] 2. An image encoding method comprising the following steps: obtaining quantized potential values ​​and tile partitioning parameters; and encoding the potential values ​​and the partitioning parameters into a bit stream.

[0367] 3. An apparatus for processing video data, comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of solutions 1-2.

[0368] 4. A non-transitory computer-readable medium, comprising a computer program product used by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, so that when the computer executable instructions are executed by a processor, the video codec device performs any method of solutions 1-2.

[0369] 5. A non-transitory computer-readable recording medium storing a bit stream of a video, the bit stream of the video being generated by a method performed by a video processing device, wherein the method includes the method of any one of solutions 1-2.

[0370] 6. A method for storing a bit stream of a video, comprising the method of any one of solutions 1-2.

[0371] 7. A method, apparatus or system as described in this document.

[0372] Another list of some example preferred solutions is provided next.

[0373] 1. An image decoding method, comprising:

[0374] Perform entropy decoding process to obtain quantized super potential samples and quantized residual potential samples Apply the potential sample prediction process to quantize the super potential samples and quantized residual potential samples Get quantified potential samples and applying a synthetic transformation process to use quantized latent samples Generate a reconstructed image.

[0375] 2. The method of solution 1, wherein performing an entropy decoding process comprises parsing two independent bitstreams contained in a single file, and wherein a first bitstream of the two independent bitstreams is decoded using a fixed probability density model.

[0376] 3. The method according to solution 2 further includes using a discrete cumulative distribution function to analyze and quantify the super potential sample points And use the super prior variance decoder to process the quantized super potential samples The hyper-prior variance decoder is a neural network (NN) based sub-network for generating the Gaussian variance σ.

[0377] 4. The method according to solution 3 further comprises applying arithmetic decoding to the second bit stream of the two independent bit streams to obtain quantized residual potential samples and assume a zero-mean Gaussian distribution

[0378] 5. The method according to any one of solutions 1-4, further comprising: An inverse transform operation is performed, and wherein the inverse transform operation is performed by a super a priori variance decoder.

[0379] 6. A method according to any one of solutions 1-5, wherein the output of the inverse transform operation is cascaded with the output of the context model module to generate a cascaded output, wherein the cascaded output is processed by a prediction fusion model to generate a prediction sample μ, and wherein the prediction sample is added to the quantized residual potential sample To obtain the potential sample points for quantification

[0380] 7. A method according to any one of solutions 1-6, wherein the potential sample prediction process is an autoregressive process.

[0381] 8. A method according to any one of solutions 1-7, wherein the quantized potential samples in different rows are processed in parallel.

[0382] 9. An image encoding method, comprising: using an analysis transform to transform an input image into a potential sample y; using a super encoder to quantize the potential sample y to generate a quantized super potential sample Use entropy coding to convert the quantized super potential samples Encoded into a bitstream; applying a latent sample prediction process to use quantized super latent samples Get quantized potential samples based on potential samples y and quantized residual potential samples After the potential sample prediction process, the predicted sample μ is obtained; and the quantized super potential sample and quantized residual potential samples Entropy encoding into a bit stream.

[0383] 10. The method according to any one of solutions 1-9 also includes rounding the output of the super encoder.

[0384] 11. A method according to any one of solutions 1-10, wherein the Gaussian variance variable σ obtained as the output of the super-prior variance decoder is used to quantize the residual potential samples Perform entropy coding.

[0385] 12. A method according to any one of solutions 1-11, wherein encoder configuration parameters are pre-optimized.

[0386] 13. A method according to any one of solutions 1-12, wherein the method is implemented by an encoder, and wherein the prepare_weights() function of the encoder is configured to calculate default pre-optimized encoder configuration parameters.

[0387] 14. A method according to any of solutions 1-13, wherein the encoder's write_weights() function includes default pre-optimized encoder configuration parameters in the high-level syntax of the bitstream.

[0388] 15. The method according to any of solutions 1-14, wherein no rate-distortion optimization process is performed.

[0389] 16. A method according to any of solutions 1-15, wherein the decoding process is not performed as part of the image encoding method.

[0390] 17. A method according to any of solutions 1-16, comprising using neural network-based adaptive image and video compression as disclosed herein.

[0391] 18. An apparatus for processing video data, comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of solutions 1-17.

[0392] 19. A non-transitory computer-readable medium, comprising a computer program product used by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, so that when the computer executable instructions are executed by a processor, the video codec device performs any method of solutions 1-17.

[0393] 20. A non-transitory computer-readable recording medium storing a bit stream of a video, the bit stream of the video being generated by a method performed by a video processing device, wherein the method comprises the method of any one of solutions 1-17.

[0394] 21. A method for storing a bit stream of a video, comprising the method of any one of solutions 1-17.

[0395] 22. A method, apparatus or system as described in this document.

[0396] In the solution described herein, an encoder can comply with the format rules by generating an encoded representation according to the format rules. In the solution described herein, a decoder can parse the syntax elements in the encoded representation using the format rules to generate decoded video according to the format rules with or without knowledge of the syntax elements.

[0397] In this document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during the conversion from a pixel representation of a video to a corresponding bitstream representation, or vice versa. For example, the bitstream representation of a current video block may correspond to bits that are co-located or distributed at different locations within the bitstream, as defined by the syntax. For example, a macroblock may be encoded based on transformed and encoded error residual values, and may also use bits in the header and other fields in the bitstream. In addition, during conversion, the decoder may parse the bitstream with the knowledge that certain fields are present or absent based on the determinations described in the above solution. Similarly, the encoder may determine whether to include certain syntax fields and generate the encoded representation accordingly by including or excluding the syntax fields in the encoded representation.

[0398] The disclosed and other solutions, examples, embodiments, modules and functional operations described in this document may be implemented in digital electronic circuits, or in computer software, firmware or hardware including the structures disclosed in this document and their structural equivalents or a combination of one or more of them. The disclosed and other embodiments may be implemented as one or more computer program products, i.e., one or more computer program instruction modules encoded on a computer-readable medium, for execution by a data processing device or for controlling the operation of the data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material composition that affects a machine-readable propagated signal, or a combination of one or more of them. The term "data processing device" encompasses all devices, equipment and machines for processing data, including, by way of example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, a device may also include code that creates an execution environment for the computer program in question, for example, code constituting a processor firmware, a protocol stack, a database management system, an operating system or a combination of one or more of them. A propagated signal is an artificially generated signal, for example, a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0399] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored as part of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a communications network.

[0400] The processes and logic flows described in this document may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may also be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0401] By way of example, processors suitable for executing computer programs include both general and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data or be operably coupled to these mass storage devices to receive data from them or transfer data to them, or both. However, a computer does not necessarily require such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and compact disk read-only memory (CD ROM) and digital versatile disk read-only memory (DVD-ROM) disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0402] Although this patent document contains many details, these details should not be interpreted as limitations on any subject matter or the scope of what may be claimed, but rather as descriptions of features that are unique to specific embodiments of specific technologies. Certain features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. In addition, although the above features may be described as working in certain combinations and even initially claimed as such, in some cases one or more features in the combination may be excluded from the claimed combination, and the claimed combination may point to a subcombination or a variation of a subcombination.

[0403] Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order or sequence shown, or that all illustrated operations be performed to achieve the desired results. In addition, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0404] Only a few implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and illustrated in this patent document.

[0405] A first component is directly coupled to a second component when there are no intervening components between the first component and the second component other than a line, trace, or another medium. A first component is indirectly coupled to a second component when there are intervening components between the first component and the second component other than a line, trace, or another medium. The term "coupled" and variations thereof include both direct coupling and indirect coupling. Unless otherwise indicated, the use of the term "approximately" is intended to include a range of ±10% of the subsequent number.

[0406] Although several embodiments have been provided in the present disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples should be considered illustrative rather than restrictive, and are not intended to be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0407] In addition, without departing from the scope of the present disclosure, the techniques, systems, subsystems, and methods described and illustrated as discrete or separate in various embodiments may be combined or integrated with other systems, modules, techniques, or methods. Other items shown or discussed as coupled may be directly connected or may be indirectly coupled or communicated through some interface, device, or intermediate component, whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and modifications may be determined and made by those skilled in the art without departing from the spirit and scope of the present disclosure.

[0408] The solutions listed in this disclosure may be used to compress images, compress videos, compress parts of images, or compress parts of videos.

[0409] Furthermore, the techniques, systems, subsystems, and methods described and illustrated in the various embodiments may be used to compress images, compress videos, compress portions of images, or compress portions of videos.

Claims

1. A method for processing image or video data in a neural network, comprising: Obtaining quantified super potential samples Quantized residual potential samples and quantified potential samples as well as Based on the quantified super potential sample The quantized residual potential samples and the potential sample points of the quantization Performs conversion between visual media data and bitstreams.

2. The method according to claim 1, further comprising: Receive a bitstream including a header, The header includes a model identifier model_id, a metric and / or a quality, wherein the metric specifies the model used in the conversion, and the quality specifies the quality of the pre-trained model.

3. The method according to claim 1 or 2, wherein: The header specifies the height of the output picture original_size_h in number of luma samples, and / or the header specifies the width of the output picture original_size_w in number of luma samples.

4. The method according to any one of claims 1 to 3, wherein: The header specifies the height resized_size_h of the reconstructed picture after the composition transform and before the resampling process in number of luma samples, and / or the header specifies the width resized_size_w of the reconstructed picture after the composition transform and before the resampling process in number of luma samples.

5. The method according to any one of claims 1 to 4, wherein: The header specifies the height latent_code_shape_h of the quantized residual latent value and / or the width latent_code_shape_w of the quantized residual latent value.

6. The method according to any one of claims 1 to 5, wherein: The header specifies the output bit depth output_bit_depth of the output reconstructed picture and / or the number of bits output_bit_shift required to be shifted when obtaining the output reconstructed picture.

7. The method according to any one of claims 1 to 6, wherein: The header specifies a double precision processing flag double_precision_processing_flag, which specifies whether double precision processing is enabled.

8. The method according to any one of claims 1 to 7, wherein: The header specifies whether deterministic processing is applied when performing the conversion between the visual media data and the bitstream.

9. The method according to any one of claims 1 to 8, wherein: The header specifies a fast resize flag fast_resize_flag, which specifies whether to use fast resize.

10. The method according to any one of claims 1 to 9, wherein: The resampling process is performed according to the fast resizing flag.

11. The method according to any one of claims 1 to 10, wherein: The header specifies a number for specifying the number of tiles, num_second_level_tile or num_first_level_tile.

12. The method according to any one of claims 1 to 11, wherein: The numbers specify the first level tile num_first_level_tile and / or the second level tile num_second_level_tile.

13. The method according to any one of claims 1 to 12, wherein: Depending on the number of slices, a composition transform or a portion of a composition transform is performed.

14. The method according to any one of claims 1 to 13, wherein: The header specifies the number of threads to use in wavefront processing, num_wavefront_max or num_wavefront_min.

15. The method according to any one of claims 1 to 14, wherein: The header specifies the maximum number of threads num_wavefront_max and / or the minimum number of threads num_wavefront_min used in wavefront processing.

16. The method according to any one of claims 1 to 15, wherein: The header specifies the number of samples in each row that are shifted, waveshift, compared to the samples in the previous row.

17. The method according to any one of claims 1 to 16, wherein: The header specifies the number of parameter sets or filters used in the adaptive quantization process to control the quantization of the residual.

18. The method according to any one of claims 1 to 17, wherein: The header includes parameters that specify how many times the adaptive quantization process is performed.

19. The method according to any one of claims 1 to 18, wherein: The adaptive quantization process is to modify the residual samples and / or variance sample (σ) process.

20. The method according to any one of claims 1 to 19, wherein: The header specifies the number of filters or parameter sets used in the residual sample skipping process.

21. The method according to any one of claims 1 to 20, wherein: The header specifies the number of parameter sets used in the latent domain masking and scaling to determine the latent samples in the quantization Scaling at the decoder after being reconstructed.

22. The method according to any one of claims 1 to 21, wherein: The header specifies the number of parameter sets used in latent domain masking and scaling to modify the quantized latent samples before applying the synthesis transform 23. The method according to any one of claims 1 to 22, wherein: The header specifies whether a thresholding operation above or below a threshold is applied in the adaptive quantization process.

24. The method according to any one of claims 1 to 23, wherein: The header specifies the value of the multiplier to be used in the adaptive quantization process or the sample skipping process or potential scaling process before synthesis.

25. The method according to any one of claims 1 to 24, wherein: The header specifies the value of the threshold to be used in the adaptive quantization process or the sample skipping process or potential scaling process before synthesis.

26. The method according to any one of claims 1 to 24, wherein: The header includes a parameter that specifies a multiplier, threshold, or number of greater than flags specified in the header.

27. The method according to any one of claims 1 to 24, wherein: The header specifies a number of parameter sets, wherein a parameter set includes a threshold parameter and a multiplier parameter.

28. The method according to any one of claims 1 to 27, wherein: The header includes an adaptive offset enable flag that specifies whether adaptive offset is used.

29. The method according to any one of claims 1 to 28, wherein: The header specifies the number of horizontal splits num_horizontal_split in the adaptive offset process and the number of vertical splits num_vertical_split in the adaptive quantization process.

30. The method according to any one of claims 1 to 29, wherein: The header specifies an offset precision offsetPrecision, and wherein a plurality of adaptive offset coefficients are multiplied by the offset precision and rounded to the nearest integer before encoding.

31. The method according to any one of claims 1 to 30, wherein: The header specifies an offset precision offsetPrecision, and wherein the adaptive offset coefficients are modified according to the offset precision.

32. The method according to any one of claims 1 to 31, further comprising: An entropy decoding process is performed, the entropy decoding process comprising parsing two independent bitstreams, and wherein a first bitstream of the two independent bitstreams is decoded using a fixed probability density model.

33. The method according to any one of claims 1 to 32, further comprising: The quantified super potential samples are analyzed using discrete cumulative distribution functions and processing the quantized super potential samples using a super a priori variance decoder The super prior variance decoder is a sub-network based on a neural network NN for generating a Gaussian variance σ.

34. The method of claim 33, further comprising: Applying arithmetic decoding to the second bit stream of the two independent bit streams to obtain the quantized residual potential samples and assume a zero-mean Gaussian distribution 35. The method according to any one of claims 1 to 34, further comprising: The super potential sample points of the quantization An inverse transform operation is performed, and wherein the inverse transform operation is performed by the super a priori variance decoder.

36. The method according to any one of claims 1 to 35, wherein: The output of the inverse transform operation is concatenated with the output of the context model module to generate a concatenated output, wherein the concatenated output is processed by a prediction fusion model to generate a prediction sample μ, and wherein the prediction sample is added to the quantized residual potential sample To obtain the potential sample points of the quantization 37. The method according to any one of claims 1 to 36, wherein: The potential sample prediction process is an autoregressive process.

38. The method according to any one of claims 1 to 37, wherein: Potential samples of the quantization in different rows are processed in parallel.

39. The method according to any one of claims 1 to 38, further comprising: The output of the super encoder is rounded.

40. The method according to any one of claims 1 to 39, wherein: The quantized residual potential samples are quantized using the Gaussian variance variable σ obtained as the output of the super-prior variance decoder Perform entropy coding.

41. The method according to any one of claims 1 to 40, wherein: The encoder configuration parameters are pre-optimized.

42. The method according to any one of claims 1 to 41, wherein: The method is performed by an encoder, and wherein a prepare_weights() function of the encoder is configured to calculate default pre-optimized encoder configuration parameters.

43. The method according to any one of claims 1 to 42, wherein: The encoder's write_weights() function includes the default pre-optimized encoder configuration parameters in the high-level syntax of the bitstream.

44. The method according to any one of claims 1 to 43, wherein: No rate-distortion optimization process is performed.

45. The method according to any one of claims 1 to 44, wherein: The decoding process is not performed as part of the encoding method.

46. ​​The method according to any one of claims 1 to 45, comprising: Use neural network based adaptive image and video compression as disclosed herein.

47. The method according to any one of claims 1 to 46, wherein: The header includes a picture header, a slice header or a partition header.

48. An apparatus for processing video data, comprising: processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1-47.

49. A non-transitory computer-readable medium, comprising a computer program product used by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, so that when the computer executable instructions are executed by a processor, the video codec device performs the method as described in any one of claims 1-47.

50. A non-transitory computer-readable recording medium, wherein: The non-transitory computer-readable recording medium stores a bit stream of a video, the bit stream of the video being generated by a method performed by a video processing device, wherein the method includes the method of any one of claims 1-47.

51. A method for storing a bitstream of a video, comprising the method as claimed in any one of claims 1-47.

52. A method, apparatus or system as described in this document.

53. An image decoding method, comprising: Perform entropy decoding process to obtain quantized super potential samples and quantized residual potential samples Apply a potential sample prediction process to obtain the quantized super potential samples from the and the quantized residual potential samples Get quantified potential samples as well as Apply a synthetic transformation process to use the quantized potential samples Generate a reconstructed image.

54. An image encoding method, comprising: Transform the input image into potential samples y using an analytical transform; The potential sample y is quantized using a super encoder to generate a quantized super potential sample The quantized super potential samples are encoded using entropy coding. Encoded as a bitstream; Applying a potential sample prediction process to use the quantized super potential samples Obtain quantized potential sample points based on the potential sample points y and quantized residual potential samples Obtaining a predicted sample point μ after the potential sample point prediction process; as well as The quantified super potential sample and the quantized residual potential samples Entropy encoding into the bit stream.

55. The method of claim 54, wherein: The picture header specifies the height original_size_h of the original input picture before the resampling process in number of luma samples, and the picture header specifies the width original_size_w of the original input picture before the resampling process in number of luma samples.

Images