Method, device and medium for visual data processing

Abstract

An embodiment of the present disclosure provides a solution for visual data processing. A method for visual data processing is proposed. The method includes obtaining an intermediate representation of visual data for conversion between visual data and a bitstream of visual data, the intermediate representation being different from a quantized latent representation of the visual data and generated based on at least one of at least one parameter, at least a portion of the quantized latent representation, a prediction of at least a portion of the quantized latent representation, or a difference between the prediction and at least a portion of the quantized latent representation, and performing a synthesis transform on the intermediate representation for conversion, the quantized latent representation being generated based on applying a first neural network to the visual data.

Description

[Technical field]

[0001] TECHNICAL FIELD Embodiments of the present disclosure relate generally to visual data processing techniques, and more particularly to neural network-based visual data encoding. [Background technology]

[0002] In the past decade, deep learning has developed rapidly in various fields, especially in computer vision and image processing. Neural networks were originally invented by interdisciplinary research of neuroscience and mathematics. It has shown powerful abilities in terms of nonlinear transformation and classification. Neural network-based image / video compression technology has made great progress in the past five years. It has been reported that the latest neural network-based image compression algorithms have achieved rate-distortion (RD) performance comparable to that of Versatile Video Coding (VVC). As the performance of neural image compression is continuously improved, neural network-based video compression has become an actively developing research field. However, the coding quality of neural network-based image / video coding is generally expected to be further improved. Summary of the Invention

[0003] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure provide a solution for visual data processing.

[0004] In a first aspect, a method for visual data processing is proposed, the method comprising: obtaining an intermediate representation of visual data for conversion between the visual data and a bitstream of visual data, the intermediate representation being different from a quantized latent representation of the visual data and being generated based on at least one of at least one parameter, at least a portion of the quantization, a prediction of at least a portion of the quantization, or a difference between the prediction and at least a portion of the quantization, and performing a synthesis transform on the intermediate representation for conversion, the quantized latent representation being generated based on applying a first neural network to the visual data.

[0005] According to the method according to the first aspect of the present disclosure, an intermediate representation different from the quantized latent representation of visual data is generated and used for synthesis transformation. Compared to the conventional solution in which the quantized latent representation is directly used for synthesis transformation, the proposed method can at least partially remove the artifacts caused by the conventional transformation process, and thus the reconstructed image can be more visually pleasing. This allows the proposed method to advantageously improve the coding quality.

[0006] In a second aspect, an apparatus for visual data processing is proposed, the apparatus comprising a processor and a non-transitory memory having instructions thereon, the instructions, when executed by the processor, causing the processor to perform a method according to the first aspect of the present disclosure.

[0007] In a third aspect, a non-transitory computer-readable storage medium is proposed. The non-transitory computer-readable storage medium stores instructions for causing a processor to perform the method according to the first aspect of the present disclosure.

[0008] In a fourth aspect, another non-transitory computer-readable storage medium is proposed, which stores a bitstream of visual data generated by a method executed by an apparatus for visual data processing, the method comprising: obtaining an intermediate representation of the visual data, the intermediate representation being different from a quantized latent representation of the visual data and being generated based on at least one of at least one parameter, at least a portion of the quantization, a prediction of at least a portion of the quantization, or a difference between the prediction and at least a portion of the quantization, and generating a bitstream based on a synthesis transformation on the intermediate representation, the quantized latent representation being generated based on applying a first neural network to the visual data.

[0009] In a fifth aspect, a method for storing a bitstream of visual data is proposed, the method comprising: obtaining an intermediate representation of the visual data, the intermediate representation being different from a quantized latent representation of the visual data and being generated based on at least one of at least one parameter, at least a portion of the quantization, a prediction of at least a portion of the quantization, or a difference between the prediction and at least a portion of the quantization, generating a bitstream based on a synthesis transformation on the intermediate representation, and storing the bitstream on a non-transitory computer-readable recording medium, the quantized latent representation being generated based on applying a first neural network to the visual data.

[0010] This Summary is provided to introduce in a simplified form some concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. [Brief description of the drawings]

[0011] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become more apparent through the following detailed description taken in conjunction with the accompanying drawings, in which like reference numerals generally refer to like components.

[0012] [Figure 1] FIG. 1 shows a block diagram illustrating an example visual data encoding system according to some embodiments of the present disclosure. [Diagram 2] A typical transform coding method is shown below. [Diagram 3] 1 shows an image from the Kodak dataset and different representations of that image. [Figure 4] We present an autoencoder network architecture that implements a hyperprior model. [Diagram 5] A block diagram of a composite model is shown. [Figure 6] 1 shows the encoding process of a composite model. [Figure 7] 4 shows a decoding process for a composite model. [Figure 8] 4 illustrates an alternative encoding process. [Figure 9] 4 illustrates an alternative decoding process. [Figure 10] Two possible implementations of the entropy coding and quantization process are shown. [Figure 11] Here is a decoder implementation that corresponds to the encoder implementation. [Figure 12] 1 illustrates an example decoder implementation according to some embodiments of the present disclosure. [Figure 13] 1 illustrates another example decoder implementation according to some embodiments of the present disclosure. [Figure 14] 1 illustrates a further exemplary decoder implementation according to some embodiments of the present disclosure. [Figure 15] 1 illustrates an example visual data decoding process according to some embodiments of the present disclosure. [Figure 16] 1 illustrates another example visual data decoding process according to some embodiments of the present disclosure. [Figure 17] 1 illustrates a flowchart of a method for visual data processing according to some embodiments of the present disclosure.

[0013] [Figure 18] FIG. 1 illustrates a block diagram of a computing device in which various embodiments of the present disclosure may be implemented.

[0014] Throughout the drawings, the same or similar reference numbers typically refer to the same or similar elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Next, the principles of the present disclosure will be described with reference to some embodiments. It should be understood that these embodiments are described for illustrative purposes only, to help those skilled in the art understand and practice the present disclosure, and do not imply any limitation on the scope of the present disclosure. The disclosure described herein can be implemented in various forms other than those described below.

[0016] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0017] References in this disclosure to "one embodiment," "an embodiment," "an exemplary embodiment," and the like indicate that the described embodiment may include a particular feature, structure, or characteristic, but not all embodiments need to include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an exemplary embodiment, it is submitted that it is within the knowledge of one of ordinary skill in the art to affect such feature, structure, or characteristic in connection with other embodiments, whether or not explicitly described.

[0018] In this specification, terms such as "first" and "second" may be used to describe various elements, but it should be understood that these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0019] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It is further understood that as used herein, the terms "comprises," "comprising," "has," "having," "includes," and / or "including" specify the presence of stated features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0020] Example Environment 1 is a block diagram illustrating an example visual data encoding system 100 that may utilize the techniques of this disclosure. As shown, the visual data encoding system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a visual data encoding device, and the destination device 120 may also be referred to as a visual data decoding device. In operation, the source device 110 may be configured to generate encoded visual data, and the destination device 120 may be configured to decode the encoded visual data generated by the source device 110. The source device 110 may include a visual data source 112, a visual data encoder 114, and an input / output (I / O) interface 116.

[0021] Visual data sources 112 may include sources such as visual data capture devices, examples of which include, but are not limited to, an interface for receiving visual data from a visual data provider, a computer graphics system for generating visual data, and / or combinations thereof.

[0022] The visual data may include one or more pictures, or one or more images, of a video. The visual data encoder 114 encodes the visual data from the visual data source 112 to generate a bitstream. The bitstream may include a bit sequence that forms an encoded representation of the visual data. The bitstream may include an encoded picture and associated visual data. An encoded picture is an encoded representation of a picture. The associated visual data may include a sequence parameter set, a picture parameter set, and other syntax structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The encoded visual data may be transmitted directly to the destination device 120 via the I / O interface 116 through the network 130A. The encoded visual data may also be stored in a storage medium / server 130B for access by the destination device 120.

[0023] The destination device 120 may include an I / O interface 126, a visual data decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain the encoded visual data from the source device 110 or the storage medium / server 130B. The visual data decoder 124 may decode the encoded visual data. The display device 122 may display the decoded visual data to a user. The display device 122 may be integrated with the destination device 120 or may be external to the destination device 120 configured to interface with an external display device.

[0024] The visual data encoder 114 and the visual data decoder 124 may operate in accordance with a visual data encoding standard, such as a video encoding standard or a still image encoding standard, and other current and / or further standards.

[0025] Some exemplary embodiments of the present disclosure are described in detail below. It should be understood that section headings are used herein for ease of understanding and are not intended to limit the embodiments disclosed in a section to that section alone. Furthermore, although certain embodiments are described with reference to Versatile Video Coding or other specific visual data codecs, the disclosed techniques are applicable to other encoding techniques. Furthermore, although some embodiments describe an encoding step in detail, it should be understood that the corresponding step of decoding that undoes the encoding is performed by a decoder. Furthermore, the term visual data processing encompasses visual data encoding or compression, visual data decoding or decompression, and visual data transcoding, where visual data is represented from one compressed format to another compressed format or at a different compressed bit rate.

[0026] 1. Overview A neural network based image and video compression method including an autoregressive sub-network, in which a latent representation of an image is first processed by an autoregressive network, then modified according to additive or multiplicative terms, and finally processed by a synthesis network to obtain a reconstructed image.

[0027] 2. Introduction In the past decade, deep learning has been rapidly developed in various fields, especially in computer vision and image processing. Inspired by the great success of deep learning technology in the computer vision field, many researchers have shifted their attention from traditional image / video compression technology to neural image / video compression technology. Neural networks were originally invented by interdisciplinary research of neuroscience and mathematics. It has shown strong abilities in terms of nonlinear transformation and classification. Neural network-based image / video compression technology has made great progress in the past five years. It has been reported that the latest neural network-based image compression algorithm has achieved RD performance equivalent to that of Versatile Video Coding (VVC), the latest video coding standard developed by MPEG and VCEG experts and JVET (Joint Video Experts Team). As the performance of neural image compression is continuously improved, neural network-based video compression has become an actively developing research field. However, neural network-based video coding is still in its infancy due to the inherent difficulty of the problem.

[0028] 2.1. Image / Video Compression Image / video compression usually refers to computing techniques that compress images / videos into binary codes for easier storage and transmission. The binary codes may or may not support lossless reconstruction of the original image / video, called lossless and lossy compression. Most scenarios do not require lossless reconstruction, so most of the effort is spent on lossy compression. Usually, the performance of an image / video compression algorithm is evaluated from two aspects: compression ratio and reconstruction quality. The compression ratio is directly related to the number of binary codes, the fewer the better. The reconstruction quality is measured by comparing the reconstructed image / video with the original image / video, the higher the better.

[0029] Image / video compression technologies can be divided into two categories: traditional video coding methods and neural network-based video compression methods. Traditional video coding methods employ transform-based solutions, where researchers exploit statistical dependencies in latent variables (e.g., DCT and wavelet coefficients) by carefully hand-designing entropy codes that model the dependencies in the quantization regime. There are two types of neural network-based video compression: neural network-based coding tools and end-to-end neural network-based video compression. The former is simply embedded in existing traditional video codecs as a coding tool and functions as part of the framework, while the latter is an independent framework developed based on neural networks without relying on traditional video codecs. Over the past 30 years, a series of traditional video coding standards have been developed to adapt to the increasing visual content. The international standardization organization ISO / IEC has two expert groups, JPEG (Joint Photographic Experts Group) and MPEG (Moving Picture Experts Group), and ITU-T also has VCEG (Video Coding Experts Group) with the aim of standardizing image / video coding technologies. The leading video coding standards issued by these organizations include JPEG, JPEG2000, H.262, H.264 / AVC, and H.265 / HEVC. After H.265 / HEVC, the Joint Video Experts Team (JVET), formed by MPEG and VCEG, has been working on a new video coding standard, Versatile Video Coding (VVC). The first version of VVC was released in July 2020. VVC has been reported to achieve an average 50% bitrate reduction at the same visual quality compared to HEVC.

[0030] Neural network-based image / video compression is not a new invention, as there are many researchers working on neural network-based image coding. However, the network architectures were relatively shallow and the performance was not satisfactory. With the support of abundant data and powerful computing resources, neural network-based methods are more commonly used in various applications. Currently, neural network-based image / video compression has shown promising improvements and its feasibility has been confirmed. However, the technology is still far from mature and many challenges need to be addressed.

[0031] Neural Networks Neural networks, also known as artificial neural networks (ANNs), are computational models used in machine learning techniques, usually consisting of multiple processing layers, each layer consisting of multiple simple but nonlinear elementary computational units. One advantage of such deep networks is considered to be that they allow processing data at multiple levels of abstraction and transforming data into different kinds of representations. It should be noted that these representations are not designed manually, but on the contrary, deep networks, including the processing layers, are learned from vast amounts of data using common machine learning procedures. Deep learning is considered to be particularly useful for processing native unstructured data, such as acoustic and visual signals, since it eliminates the need for manual representations, but processing such data has been a long-standing challenge in the field of artificial intelligence.

[0032] 2.3. Neural Networks for Image Compression Existing neural networks for image compression methods can be divided into two categories: pixel probability modeling and autoencoders. The former belong to predictive coding strategies, while the latter are transform-based solutions. Sometimes, these two methods are combined together in the literature.

[0033] 2.3.1. Pixel Probability Modeling According to Shannon's information theory, the optimal method for lossless coding can reach the minimum code rate -log2p(x), where p(x) is the probability of symbol x. Many lossless coding methods have been developed in the literature, among which arithmetic coding is considered to be one of the optimal methods. Given a probability distribution p(x), arithmetic coding ensures that the code rate is as close as possible to the theoretical limit of -log2p(x) without considering the rounding error. Therefore, the remaining problem is how to determine the probability, which is very difficult for natural images / videos due to the curse of dimensionality.

[0034] According to the predictive coding strategy, one way to model p(x) is to predict pixel probabilities one by one in raster scan order based on previous observations, where x is the image. p(x)=p(x1)p(x2|x1)…p(x i |x1,…,x i-1 )…p(x m×n |x1,…,x( m×n-1 ) (1) where m and n are the height and width of the image, respectively. The previous observations are also known as the context of the current pixel. When the image is large, it is difficult to estimate the conditional probability, so a convenient way is to bound the scope of the context. p(x)=p(x1)p(x2|x1)…p(x i |x i-k ,…,x i-1 )…p(x m×n |x m×n-k ,…,x m×n-1 ) (2) where k is a predefined constant that controls the scope of the context.

[0035] It should be noted that this condition may also take into account the sample values ​​of other color components. For example, when encoding RGB color components, the R sample depends on the previously encoded pixel (including R / G / B samples), the current G sample is encoded according to the previously encoded pixel and the current R sample, and when encoding the current B sample, the previously encoded pixel and the current R and G samples may also be taken into account.

[0036] Neural networks were originally introduced for computer vision tasks and have proven to be effective in regression and classification problems. Therefore, we use a neural network to train a set of vectors of images from a set of contexts x1, x2, …, x i-1 Given p(x i In [7], it is proposed to estimate the probability of a binary image, i.e., x i A pixel probability is proposed for ∈{-1,+1}. The Neural Autoregressive Distribution Estimator (NADE) is designed for pixel probability modeling and is a feedforward network with a single hidden layer. There are similar studies in existing designs, where the feedforward network also has connections that skip the hidden layer and parameters are shared. Experiments are performed on the binarized MNIST dataset. In the existing design, NADE is extended to a real-valued model RNADE, which estimates the probability p(x i |x1,…,x i-1 ) is derived from a Gaussian mixture. Their feedforward network also has a single hidden layer, but the hidden layer is resized to avoid saturation and uses rectified linear units (ReLU) instead of sigmoids. On existing designs, NADE and RNADE improve by rearranging the pixel order and using deeper neural networks.

[0037] Designing advanced neural networks plays a key role in improving pixel probability modeling. In the existing design, a multidimensional long short-term memory (LSTM) is proposed, where LSTM is coupled with conditional Gaussian scale mixtures for probability modeling. LSTM is a special recurrent neural network (RNN) that has proven to be suitable for modeling sequential data. A spatial variant of LSTM is used for images later in the existing design. Several different neural networks have been studied, including RNNs and CNNs, namely PixelRNN and PixelCNN, respectively. In PixelRNN, two variants of LSTM, called row LSTM and diagonal BiLSTM, are proposed, the latter of which is specifically designed for images. PixelRNN incorporates residual connections that help in training deep neural networks with up to 12 layers. In PixelCNN, masked convolutions are used to fit the shape of the context. Compared with previous studies, PixelRNN and PixelCNN are more specialized for natural images: they consider pixels as discrete values ​​(e.g., 0, 1, ..., 255) and predict multinomial distributions over the discrete values; they handle color images in RGB color space; and they work well on the large-scale image dataset ImageNet. In the existing design, GatedPixelCNN is proposed to improve PixelCNN, achieving comparable performance to PixelRNN while significantly reducing the complexity. In the existing design, PixelCNN++ proposes the following improvements over PixelCNN: using discrete logistic mixture likelihood instead of 256-way multinomial distribution, using downsampling to capture structures at multiple resolutions, introducing additional shortcut connections to speed up training, adopting dropout for regularization, and combining RGB into one pixel. In the existing design, PixelSNAIL is proposed, which combines casual convolution and self-attention.

[0038] Most of the above methods directly model the probability distribution in the pixel domain. Other researchers try to model the probability distribution as a conditional model based on explicit or latent representations. That is, we may estimate

[0039]

number

[0040] Autoencoder Autoencoders originate from the famous work proposed by Hinton and Salakhutdinov. This method is trained for dimensionality reduction and consists of two parts: encoding and decoding. The encoding part converts a high-dimensional input signal into a lower-dimensional representation, which generally has a smaller spatial size but a larger number of channels. The decoding part tries to recover the high-dimensional input from the lower-dimensional representation. Autoencoders allow automatic learning of representations, eliminating the need to manually create features, which is also considered one of the most important advantages of neural networks.

[0041] Figure 2 shows a typical transform coding scheme. An original image x is fed to a parsing network g to achieve a latent representation y. a The latent representation y is then quantized and compressed into bits. The number of bits R is used to measure the coding rate. The quantized latent representation (outside 1) TIFF2025506992000003.tif18161 (hereinafter referred to as "y^") is a composite network g s The reconstructed image x^ is obtained by inverse transformation using the function g p It is calculated by converting x and x^ in

[0042] Applying autoencoder networks to lossy image compression is intuitive. It only requires encoding the latent representation learned from a well-trained neural network. However, adapting autoencoders to image compression is not straightforward, because the original autoencoder is not optimized for compression, so directly using a trained autoencoder is not efficient. In addition, there are other major challenges: First, the low-dimensional representation must be quantized before being encoded, but quantization is non-differentiable, which is necessary in backpropagation when training a neural network. Second, the objectives are different in compression scenarios, since both distortion and rate must be considered. Rate estimation is difficult. Third, a practical image coding scheme needs to support variable rates, scalability, encoding / decoding speed, and interoperability. In response to these challenges, many researchers are actively contributing to the field.

[0043] In Fig. 2, the prototype of an autoencoder for image compression can be viewed as a transform coding strategy. The original image x is fed to an analytic network y=g a (x), where y is the latent representation to be quantized and encoded. The synthesis network inverse transforms the quantized latent representation ŷ to obtain the reconstructed image x̂=g s We obtain (y^). The framework is trained with a rate-distortion loss function, i.e., L=D+λR, where D is the distortion between x and x^, R is the rate calculated or estimated from the quantized representation y^, and λ is the Lagrange multiplier. It should be noted that D can be calculated in either the pixel domain or the perceptual domain. All existing research results follow this prototype, differing only in the network structure and loss function.

[0044] In terms of network structure, RNN and CNN are the most widely used architectures. In the RNN-related category, Toderici et al. propose a general framework for variable-rate image compression using RNNs. They use binary quantization to generate codes and do not consider the rate during training. This framework certainly provides scalable coding capabilities, and RNNs with convolutional and deconvolutional layers are reported to perform adequately. proposed an improved version that upgrades the encoder with a neural network similar to PixelRNN to compress binary codes. Its performance is reported to be better than JPEG on the Kodak image dataset using the MS-SSIM evaluation metric. Johnston et al. further improve the RNN-based solution by introducing hidden state priming. In addition, a SSIM weighting loss function is also designed to enable a spatially adaptive bitrate mechanism. They use MS-SSIM as the evaluation metric and achieve better results than BPG on the Kodak image dataset. Covell et al. support spatially adaptive bitrate by training a stop-code tolerant RNN.

[0045] (outside 2) TIFF2025506992000004.tif16161 (hereinafter also referred to as "Balle") et al. propose a general framework for rate-distortion optimized image compression. They use multilevel quantization to generate integer codes and consider the rate during training, i.e., the loss is the rate-distortion joint cost, which can be MSE or other. They add random uniform noise to stimulate quantization during training, and use the differential entropy of the noise code as a proxy for the rate. They use generalized decomposition normalization (GDN) as the network structure, which consists of linear mapping and nonlinear parametric normalization. The effectiveness of GDN in image coding has been verified with existing designs. Ball? et al. then propose an improved version, which uses three convolutional layers, each followed by a downsampling layer and a GDN layer, as the forward transform. Correspondingly, they use three inverse GDN layers, each followed by an upsampling layer and a convolutional layer, to stimulate the inverse transform. An arithmetic coding method is also invented to compress the integer codes. Its performance is reported to be better than JPEG and JPEG2000 in terms of MSE on the Kodak dataset. Furthermore, Balle et al. improve this method by creating a scale hyper-prior for the autoencoder. They a Let the latent representation y be z=h a (y), and z is quantized and transmitted as side information. Thus, the inverse transformation is s This is achieved by this subnetwork s attempts to decode the quantized side information z^ into the standard deviation of the quantized y^, which is further used during the arithmetic coding of y^. On the Kodak image set, their method performs slightly worse than BPG in terms of PSNR. D. Minnen et al. further exploit the structure of the residual space by introducing an autoregressive model to estimate both the standard deviation and the mean. In the existing design, Z. Cheng et al. further remove the redundancy of the residuals using a Gaussian mixture model. The reported performance is comparable to VVC on the Kodak image set, using PSNR as the evaluation metric.

[0046] 2.3.3. Hyperprior models In the transform coding approach to image compression, the encoder subnetwork (Section 2.3.2) is a parametric analysis transform g a (x,φ g ) to convert an image vector x into a latent representation y, which is then quantized to form ŷ. Because ŷ is a discrete value, it can be losslessly compressed using entropy coding techniques such as arithmetic coding and transmitted as a bit sequence.

[0047] As is evident from the center-left and center-right images in Figure 3, there is a large spatial dependence between the elements of ŷ. In particular, their scales (center-right image) appear to be spatially coupled. In the existing design, an additional set of random variables ẑ is introduced to capture the spatial dependence and further reduce the redundancy. In this case, the image compression network is depicted in Figure 4.

[0048] In Figure 4, the left hand of the model is the encoder g a and decoder g s (explained in Section 2.3.2). The right hand side is an additional hyperencoder h a and Hyper Decoder h s In this architecture, the encoder converts an input image x into a a We obtain a response y with a spatially varying standard deviation. This response y is expressed as h a , which sums up the standard deviation distribution in z. z is then quantized (ẑ), compressed, and transmitted as side information. The encoder uses the quantized vector ẑ to estimate the spatial distribution σ of the standard deviation, and uses it to compress and transmit the quantized image representation ŷ. The decoder first recovers ẑ from the compressed signal. Then h s Using this, we can obtain σ, which gives a correct probability estimate and successfully recovers y^. Then, we can use g s and obtain the reconstructed image.

[0049] When a hyperencoder and hyperdecoder are added to an image compression network, the spatial redundancy of the quantized latent y^ is reduced. The right-most image in Figure 3 corresponds to the quantized latent when a hyperencoder / decoder is used. Compared to the center-right image, the spatial redundancy has been significantly reduced due to the lower correlation of samples in the quantized latent.

[0050] Figure 3 shows an image from the Kodak dataset and different representations of that image. The leftmost image in Figure 3 shows an image from the Kodak dataset. The center-left image in Figure 3 shows a visualization of the latent representation y of that image. The center-right image in Figure 3 shows the standard deviation σ of that latent. The rightmost image in Figure 3 shows the latent y after the introduction of a hyper-prior (hyper-encoder and decoder) network.

[0051] Figure 4 shows the network architecture of an autoencoder that implements the hyper-prior model. The left side shows the image autoencoder network, and the right side corresponds to the hyper-prior subnetwork. a and g a Q stands for quantization, and AE and AD stand for arithmetic encoder and arithmetic decoder, respectively. The hyper-prior model is a hyper-encoder (h a ) and Hyper Decoder (h s The hyper-prior model generates a quantized hyper-latent (ẑ), which contains information about the probability distribution of samples of the quantized latent ŷ. ẑ is included in the bitstream and transmitted to the receiver (decoder) together with ŷ.

[0052] 2.3.4. Context Model While the hyperprior model improves modeling the probability distribution of the quantized latent ŷ, further improvements can be obtained by utilizing an autoregressive model that predicts the quantized latent from its causal context (the context model).

[0053] The term autoregressive means that the output of a process is later used as the input to that process. For example, a context model sub-network generates one latent sample, which is later used as the input to obtain the next sample.

[0054] Existing designs utilize a joint architecture in which both the hyper-prior model sub-networks (hyper-encoder and hyper-decoder) and the context model sub-network are utilized. The hyper-prior and the context model are combined to learn a probabilistic model for the quantized latent ŷ, which is then used for entropy coding. As shown in Figure 5, the outputs of the context sub-network and the hyper-decoder sub-network are combined by a sub-network called entropy parameter, which generates the mean μ and scale (or variance) σ parameters for the Gaussian probability model. The Gaussian probability model is then used to encode the samples of the quantized latent into a bitstream with the help of an arithmetic encoder (AE) module. At the decoder, the Gaussian probability model is used to obtain the quantized latent ŷ from the bitstream by the arithmetic decoder (AD) module.

[0055] Figure 5 shows a block diagram of the combined model. The combined model jointly optimizes an autoregressive component that estimates the probability distribution of the latents from their causal context (context model) along with a hyperprior and an underlying autoencoder. The real-valued latent representation is quantized (Q) to create a quantized latent (ŷ) and a quantized hyper-latent (ẑ), which are compressed into a bitstream by an arithmetic encoder (AE) and decompressed by an arithmetic decoder (AD). The highlighted regions correspond to the components executed by the receiver (i.e., decoder) to recover the image from the compressed bitstream.

[0056] Typically, but not limited to, the latent samples are modeled as a Gaussian distribution or a Gaussian mixture model. In existing designs, according to Figure 5, a context model and a hyper-prior are jointly used to estimate the probability distribution of the latent samples. Since a Gaussian distribution can be defined by a mean and a variance (also known as sigma or scale), a joint model is used to estimate the mean and variance (denoted as μ and σ).

[0057] 2.3.5. Coding process using the autoregressive hyperprior joint model Figure 5 corresponds to the latest compression method. In this section and the next section, the encoding and decoding processes are explained respectively.

[0058] Figure 6 shows the encoding process. The input image is first processed by the encoder sub-network. The encoder converts the input image into a transformed representation called latent, denoted by y. y is fed into a quantizer block, denoted by Q, to obtain the quantized latent (y^). y^ is converted into a bitstream (bits1) using an arithmetic coding module (denoted by AE). The arithmetic coding block converts each sample of y^ sequentially, one by one, into a bitstream (bits1).

[0059] The hyperencoder, context, hyperdecoder and entropy parameter sub-network modules are used to estimate the probability distribution of samples of the quantized latent y^. The latent y is input to the hyperencoder, which outputs a hyperlatent (denoted by z). The hyperlatent is quantized (z^) and a second bitstream (bits2) is generated using the arithmetic coding (AE) module. The factorized entropy module generates a probability distribution that is used to encode the quantized hyperlatent into a bitstream. The quantized hyperlatent contains information about the probability distribution of the quantized latent (y^).

[0060] The entropy parameter sub-network generates an estimate of the probability distribution used to encode the quantized latent y^. The information generated by the entropy parameters typically includes the mean μ and scale (or variance) σ parameters that are used jointly to obtain a Gaussian probability distribution. The Gaussian distribution of a random variable x is given by (Outside 3) It is defined as TIFF2025506992000005.tif26161, where the parameter μ is the mean or expectation (and median and mode) of the distribution, and the parameter σ is its standard deviation (or variance, scale). To define a Gaussian distribution, the mean and variance need to be determined. In existing designs, an entropy parameter module is used to estimate the mean and variance values.

[0061] The sub-network hyperdecoder generates part of the information used by the entropy parameter sub-network, and the other part of the information is generated by an autoregressive module called the context module. The context module generates information about the probability distribution of the samples of the quantized latent using samples already coded by the arithmetic coding (AE) module. The quantized latent y^ is usually a matrix consisting of a number of samples. The samples can be indicated using indices like y^[i,j,k] or y^[i,j] depending on the dimension of the matrix y^. Samples y^[i,j] are usually coded one by one by the AE using a raster scan order, where the rows of the matrix are processed from top to bottom and the samples within a row are processed from left to right. In such a case (when the AE codes the samples into the bitstream using a raster scan order), the context module generates information related to the sample y^[i,j] in the raster scan order using the previously coded samples. The information generated by the context module and the hyperdecoder is combined by the entropy parameter module to generate a probability distribution that is used to encode the quantized latent ŷ into a bitstream (bits1).

[0062] Finally, the first bitstream and the second bitstream are transmitted to a decoder as a result of the encoding process.

[0063] Note that other names for the above modules are possible.

[0064] In the above explanation, all the elements in Fig. 6 are collectively called encoders. Also, the analysis transformation that converts an input image into a latent representation is called an encoder (or autoencoder).

[0065] 2.3.6. Decoding using the joint autoregressive hyperprior model Figure 7 shows the decoding process separately. In the decoding process, the decoder first receives the first bitstream (bits1) and the second bitstream (bits2) generated by the corresponding encoder. bits2 is first decoded by the arithmetic decoding (AD) module using the probability distribution generated by the factorized entropy sub-network. The factorized entropy module typically generates the probability distribution using a predefined template, e.g., a predefined mean and variance value in the case of a Gaussian distribution. The output of the arithmetic decoding process of bits2 is the quantized hyper-latent ẑ. The AD process goes back to the AE process applied in the encoder. The AE and AD processes are lossless, meaning that the quantized hyper-latent ẑ generated by the encoder can be reconstructed as is in the decoder.

[0066] After obtaining z^, it is processed by the hyperdecoder, whose output is sent to the entropy parameter module. The three sub-networks employed in the decoder, the context, the hyperdecoder and the entropy parameters, are identical to those in the encoder. Therefore, the decoder obtains the exact same probability distribution, which is essential for losslessly reconstructing the quantized latent y^. As a result, the decoder obtains the same quantized latent y^ obtained in the encoder.

[0067] After the probability distribution (e.g., mean and variance parameters) is obtained by the entropy parameter sub-network, the arithmetic decoding module decodes the quantized latent samples one by one from the bitstream bits1. From a practical point of view, the autoregressive model (context model) is inherently serial and therefore cannot be accelerated using techniques such as parallelization.

[0068] Finally, the fully reconstructed quantized latent ŷ is input to the synthesis transform (denoted as decoder in Fig. 7) module to obtain the reconstructed image.

[0069] In the above description, all elements shown in Fig. 7 are collectively referred to as a decoder. Also, the synthesis transform that converts the quantized latent image into a reconstructed image is also referred to as a decoder (or autodecoder).

[0070] 2.3.7. Alternative coding process using joint autoregressive hyperprior models An alternative implementation of the encoder can be depicted in Figure 8. It differs from the encoder depicted in Figure 6 in how the quantized latent values ​​are obtained and what is coded into the bitstream.

[0071] In Fig. 6, the samples of quantized latent ŷ are obtained by quantizing the latent y. Furthermore, the samples of ŷ are included in the bitstream by an entropy encoder (or an arithmetic encoder), where an arithmetic encoder is a special case of an entropy encoder.

[0072] In Fig. 8, the quantized latent y^ is obtained according to (e.g., by summing) the mean value μ and the quantized residual latent w^. The mean value is obtained recursively by using previously obtained samples of y^. The quantized residual latent w^ is obtained by subtracting μ from the latent y and quantizing the result. Finally, an arithmetic encoder module (AE) encodes the quantized residual latent into a bitstream.

[0073] 2.3.8. Alternative Decoding Process Using Joint Autoregressive Hyperprior Model An alternative implementation of the decoder can be depicted in Figure 9. It differs from the decoder depicted in Figure 7 in how it obtains the quantized latents and what it gets from the bitstream.

[0074] In Fig. 9, the quantized residual latency ŵ is obtained from the bitstream by an arithmetic decoder (AD, or entropy decoder). Then, the quantized latency ŷ is obtained using the mean value μ and the quantized residual ŵ (e.g., by summing them). The mean value is obtained by a prediction module using previously obtained samples of ŷ.

[0075] 2.4. Neural Networks for Video Compression As with traditional video coding techniques, neural image compression serves as the basis for intra-compression in neural network-based video compression, so the development of neural network-based video compression techniques is later than that of neural network-based image compression, but it requires much more effort to solve the challenges due to its complexity. Since 2017, a few researchers have been working on neural network-based video compression methods. Compared with image compression, video compression requires efficient methods to remove redundancy between pictures. And inter-picture prediction is a key step in these studies. Motion estimation and compensation have been widely adopted but have not been implemented by trained neural networks until recently.

[0076] Research on neural network-based video compression can be divided into two categories, depending on the scenarios they target: random access and low latency. For random access, the need exists to be able to start decoding at any point in the sequence, and typically involves splitting the entire sequence into multiple separate segments, each of which can be decoded independently. For low latency, the goal is to reduce the decoding time, and typically involves simply using a frame that is earlier in time as a reference frame to decode a later frame.

[0077] 2.4.1. Low Latency Chen et al. were the first to propose a video compression scheme using trained neural networks. They first divide a video sequence frame into blocks, and each block can be coded in one of two available modes: intra-coding or inter-coding. If intra-coding is selected, the block is compressed with an associated autoencoder. If inter-coding is selected, motion estimation and compensation are performed in a traditional manner, and the trained neural network is used for residual compression. The output of the autoencoder is directly quantized and coded by the Huffman method.

[0078] Chen et al. propose another neural network-based video coding method using PixelMotionCNN. Frames are compressed in time order, and each frame is divided into blocks that are compressed in raster-scan order. Each frame is first extrapolated with the two previous reconstructed frames. When a block is compressed, the extrapolated frame is fed to PixelMotionCNN along with the context of the current block to derive a latent representation. The residual is then compressed by a variable-rate image method. This method has performance comparable to H.264.

[0079] Lu et al. propose a real-sense end-to-end neural network-based video compression framework in which all modules are implemented with neural networks. The scheme accepts the current frame and a pre-reconstructed frame as input and derives the optical flow as motion information using a pre-trained neural network. The motion information is warped with the reference frame, and then the neural network generates a motion-compensated frame. The residual and motion information are compressed with two separate neural autoencoders. The entire framework is trained with a single rate-distortion loss function. It achieves better performance than H.264.

[0080] Rippel et al. propose an advanced neural network-based video compression method, which inherits and extends the traditional neural network-based video coding method and has the following key features: 1) it uses only one autoencoder to compress the motion information and residuals, 2) it performs motion compensation with multiple frames and multiple optical flows, and 3) it learns the online state and propagates it to the next frame over time. The method achieves better performance than the HEVC reference software in MS-SSIM.

[0081] J. Lin et al. propose an extended end-to-end neural network-based video compression framework. In their solution, multiple frames are used as references. This allows the use of multiple reference frames and associated motion information to provide a more accurate prediction of the current frame. Furthermore, motion field prediction is employed to remove motion redundancy along the time channel. A post-processing network is also introduced in this work to remove reconstruction artifacts due to pre-processing. Its performance outperforms H.265 by a notable margin in both PSNR and MS-SSIM.

[0082] Eirikur et al. propose a scale-space flow to replace the commonly used optical flow by adding a scale parameter, and it is reported to achieve better performance than H.264.

[0083] Z. Hu et al. propose a multi-resolution representation of optical flow. Specifically, a motion estimation network generates multiple optical flows with different resolutions, and the network learns which one to select under a loss function. The performance is slightly improved and outperforms H.265.

[0084] Random Access Wu et al. propose a neural network-based video compression method with frame interpolation. First, key frames are compressed with a neural image compressor, and the remaining frames are compressed in a hierarchical order. The method performs motion compensation in the perceptual domain, i.e., derives feature maps at multiple spatial scales of the original frames, and uses motion to warp the feature maps, which are then used in the image compressor. The method is reported to be equivalent to H.264.

[0085] Djelouah et al. propose an interpolation-based video compression method, in which the interpolation model combines motion information compression and image synthesis, using the same autoencoder for both the image and the residual.

[0086] propose a neural network-based video compression method based on a variational autoencoder with a deterministic encoder. Specifically, the model consists of an autoencoder and an autoregressive prior. Unlike previous methods, the method accepts a group of pictures (GOP) as input and incorporates a 3D autoregressive prior by considering temporal correlation when encoding the latent representation. It provides comparable performance to H.265.

[0087] 2.5 Introduction Natural images / videos are almost all in digital format. Grayscale digital images are (outside 4) TIFF2025506992000006.tif17161 (hereafter, "x∈D m×n ", etc.), where D is a set of pixel values, m is the height of the image, and n is the width of the image. For example, D={0,1,2,...,255} is a common setting, where |D|=256=2 8 , so a pixel can be represented by an 8-bit integer. An uncompressed grayscale digital image is 8 bits per pixel (bpp), but compressed with significantly fewer bits.

[0088] Color images are usually represented with multiple channels to record color information. For example, in the RGB color space, an image is represented as x ∈ D m×n×3 where three separate channels store red, green, and blue information. Similar to an 8-bit grayscale image, an uncompressed 8-bit RGB image is 24 bpp. Digital images / videos can be represented in different color spaces. Neural network-based video compression methods are mainly developed in RGB color space, while traditional codecs usually use YUV color space to represent video sequences. In YUV color space, an image is decomposed into three channels: Y, Cb, and Cr, where Y is the luma component and Cb / Cr are the chroma components. Because the human visual system has low sensitivity to the chroma component, Cb and Cr are generally downsampled to achieve pre-compression.

[0089] A color video sequence consists of multiple color images, called frames, that record a scene at different timestamps. For example, in the RGB color space, a color video is represented by a sequence of X = {x0, x1, …, x t ,…,x T-1}, where T is the number of frames in this video sequence, and x∈D m×n Let m=1080, n=1920, |D|=2. 8 and the video is at 50 frames per second (fps), then the data rate of this uncompressed video is 1920 x 1080 x 8 x 3 x 50 = 2,488,320,000 bits per second (bps), or about 2.32 Gbps, which requires a lot of storage, so it needs to be compressed before sending it over the Internet.

[0090] Typically, lossless methods can only achieve a compression ratio of 1.5 to 3 for natural images, which is clearly below the required level. Therefore, lossy compression methods have been developed to achieve higher compression ratios, but at the cost of distortion. Distortion can be measured by calculating the mean squared difference between the original and reconstructed images, or the mean squared error (MSE). For grayscale images, MSE can be calculated as follows:

[0091]

number

[0092]

number

[0093] To compare different lossy compression methods, it is sufficient to compare the rates obtained from the compression ratio or vice versa. However, to compare different lossy compression methods, both the rate and the reconstruction quality must be considered. For example, a commonly adopted method is to calculate the relative rates at several different quality levels and then average the rates, the average relative rate being known as the Bjontegaard delta rate (BD rate). There are other important aspects to evaluate image / video coding methods, including encoding / decoding complexity, scalability, robustness, etc.

[0094] 2.5.1. Quantization and Entropy Coding The terms quantization and entropy coding can be defined as follows: Quantization, in mathematics and digital signal processing, is the process of mapping input values ​​from a larger set (often a continuous set) to output values ​​from a (countable) smaller set (often with a finite number of elements). Rounding and truncation are classic examples of quantization processes. Quantization is involved to some degree in almost all digital signal processing, since the process of representing a signal in digital form usually involves rounding. Quantization is also at the heart of essentially all lossy compression algorithms.

[0095] The quantization formula is:

[0096]

number

[0097] Entropy coding is a common lossless data compression method that encodes symbols using an amount of bits inversely proportional to the symbol's probability. The term is used to denote several compression algorithms that operate based on the probability distribution of the source symbols. Huffman coding and arithmetic coding are two representative algorithms of this type. To perform entropy coding, the statistical properties of the symbols (e.g., their probability distribution) must be known.

[0098] Figure 10 shows two possible implementations of the entropy coding and quantization processes in NN-based image compression. The encoder's perspective is depicted.

[0099] In Fig. 10, two possible implementations of the encoder of the NN-based image compression method are shown. The encoder of the NN-based image compression method consists of four modules. The first module is an analysis transform. The analysis transform converts the input image into a latent representation y. The converted representation y is usually easier to compress than the input image because the analysis transform is designed to eliminate redundancy in the input image signal. The second module is an estimation module that estimates statistical properties of the latent representation. The statistical properties may include the mean (denoted by μ), the variance (denoted by σ) or higher moments. The third module is a quantization module that converts the continuous latent representation signal into quantized discrete values. Finally, the fourth module is an entropy coding (denoted by EC) that converts the quantized symbol values ​​into a bitstream using the statistical properties generated in the estimation module.

[0100] The encoder has two slightly different implementations depending on how the mean value μ is used. In the first implementation, the entropy coding module takes the mean value and other statistics (such as the variance σ) as input and uses them to encode the quantized latent symbol ŷ into a bitstream. In the second implementation, the mean value is first subtracted from the latent symbol y to obtain a residual sample w, which is then quantized in the quantization module to obtain a quantized residual sample ŵ. The entropy coding module uses the statistical information generated by the estimation module to convert the quantized residual sample into a bitstream. The statistical information may include, for example, the variance σ.

[0101] FIG. 11 shows a decoder implementation corresponding to the encoder implementation shown in FIG. 10. FIG. 11 illustrates two possible alternative implementations of the decoder in the NN-based video compression method. According to the first alternative implementation, an estimation module estimates the mean value μ and other statistical properties (e.g., variance σ). The variance and any other statistical information are used by an entropy coder (denoted by EC) to decode the bitstream into quantized residual samples ŵ. The estimated mean value μ and the quantized residual samples ŵ are then added together to obtain quantized latent samples ŷ. ŷ is finally transformed into a reconstructed image by a synthesis transform. The top flowchart of FIG. 11 depicts the first alternative decoder implementation.

[0102] According to a second alternative implementation, an estimation module estimates the mean μ and other statistical properties (e.g., variance σ). The mean, variance, and any other statistical information are used by an entropy coder (denoted EC) to decode the bitstream into quantized residual samples, quantized latent samples ŷ, which are finally transformed into a reconstructed image by a synthesis transform. The second alternative decoder implementation is depicted in the bottom flow chart of FIG. 11.

[0103] In the above, we denote the quantized latent samples by ŷ. The samples are not necessarily scalar values, but may be vectors and may contain multiple elements. In the remainder of this application, samples may be denoted by ŷ[i,j] or ŷ[:,i,j] or ŷ[c,i,j]. In the latter, the ":" is used to indicate the presence of a third dimension and to emphasize that the sample has multiple elements.

[0104] 3. Problems 3.1. Core issues In the state of the art, autoregressive networks are used to predict samples of the quantized latent y^. The term autoregressive denotes self-reference or self-dependence. Previously obtained samples of y^ are used to generate a prediction for obtaining the next sample of y^. This process is autoregressive because obtaining one sample of y^ depends on at least one previously obtained sample of y^.

[0105] Once all the samples of y^ are obtained, they are processed by the synthesis transform to obtain the reconstructed image. The problem with the state of the art is that the same samples of y^ are used for both the prediction module and the synthesis transform. This can be observed in Figures 7 and 9, where the same y^ is used in a loop that contains the "decoder" module (in other words the synthesis transform) and the "context" module. The "context" module is the part of the prediction module that is used to predict the next sample of y^.

[0106] Therefore, in the art, the same latent samples are used both to obtain the reconstructed image by the synthesis transform and to predict the next latent sample. However, the sample values ​​that result in a good prediction do not necessarily result in the most visually pleasing reconstructed image. For example, a subject may prefer a reconstructed image with sharper details and higher contrast. On the other hand, latent samples that result in sharper reconstructed images are generally more difficult to predict and therefore require more side information to be transmitted (increased bit rate).

[0107] 4. Detailed solutions The detailed solutions below should be considered as examples to illustrate the general concept. These solutions should not be interpreted in a narrow sense. Moreover, these solutions can be combined in any manner.

[0108] 4.1. Solution Target The goal of image compression is to maximize the quality of the output image while simultaneously reducing the number of bits to be transmitted (bit rate). This solution proposes a way to improve the quality of the reconstructed image without increasing the bit rate in neural network-based image and video compression methods.

[0109] 4.2. The Core of the Solution According to this solution, the samples of the quantized latent y^ used by the synthesis transform are different from the samples used by the prediction module. The core of the solution includes the following steps: 1. Get the first sample of the quantized latent ŷ. 2. Use the first sample to predict the second sample of ŷ (i.e., obtain a prediction). 3. Use the prediction to get the second sample of ŷ. 4. Correct the first sample and use the corrected quantized latent y s Get ^. 5. y by composition transformation s ^ is processed to obtain a reconstructed image. In other words, the quantized latent is modified after it is used by the prediction and before the synthesis transform is applied.

[0110] 4.2.1. Example implementation of the solution According to this solution, the reconstructed image is obtained by the following steps: First, we autoregressively obtain samples of the quantized latents using the following steps: 1. Obtain samples of mean μ using a prediction module. The input of the prediction module is the previously obtained samples of quantized latent ŷ. 2. Optionally, obtain samples of the residual ŵ from the bitstream. 3. Use the mean and / or residual samples to obtain new samples of the quantized latent ŷ. 4. Using the mean and / or residual samples and the quantized latent ŷ, we calculate the corrected quantized latent y s Obtain a new sample of ^. The corrected quantized latent y s After obtaining the samples of,ˆ,, we apply the synthesis transformation to obtain the reconstructed image.

[0111] 4.2.1.1. Second Option According to this solution, the reconstructed image is obtained by the following steps: First, we autoregressively obtain samples of the quantized latents using the following steps: 1. Use a prediction module to obtain samples of mean μ. The input of the prediction module is the previously obtained samples of the quantized latent ŷ. 2. Optionally, obtain samples of the residual ŵ from the bitstream. 3. Use the mean and / or residual samples to obtain new samples of the quantized latent ŷ. After taking a sample of y^, we use y^ and the mean μ (or residual w^) to obtain the corrected quantized latent y s Get ^. The synthesis transform is the modified quantized latent y s ^ is applied to obtain the reconstructed image.

[0112] 4.3. Difference between Solutions and State-of-the-Art In state-of-the-art NN-based image and video coding systems, the reconstructed image is obtained according to the following steps: First, we autoregressively obtain samples of the quantized latents using the following steps: 1. Obtain samples of mean μ using a prediction module. The input of the prediction module is the previously obtained samples of quantized latent ŷ. 2. Optionally, obtain samples of the residual ŵ from the bitstream. 3. Use the mean and / or residual samples to obtain new samples of the quantized latent ŷ. Once we have all the samples of the quantized latent ŷ, we apply the synthesis transformation to obtain the reconstructed image.

[0113] In the state of the art, the same quantized latent samples are used in the prediction module and the synthesis transform, whereas in the present solution, the modified quantized latent samples used in the synthesis transform are different from the quantized latent samples used in the prediction module.

[0114] 4.4. Solution Details In this section we present three possible implementations of the solution.

[0115] Alternative Implementation 1 FIG. 12 shows a possible implementation of a solution in NN-based image compression. It is depicted from the decoder's point of view. FIG. 12 shows one possible implementation of the solution. According to this solution, first a bitstream is received (denoted as bits1 or bits). An entropy decoder (ED) module decodes the bitstream to obtain a first sample of the quantized latent y^. The first sample is denoted as y1^. y1^ is then used by a prediction module to obtain a second sample of the statistical parameters μ2 and / or σ2. The statistical parameters may be, but are not limited to, a mean parameter μ or a variance parameter σ. The statistical parameters may also include parameters that can be used to describe a probability distribution. After the second sample of the statistical parameters is obtained, it is used by the prediction module to obtain a second sample of the quantized latent y2^. This process is repeated until all samples of the quantized latents are obtained.

[0116] As explained in the above paragraph, the first sample of the quantized latent is used by a prediction module, the output of which is used to obtain the second sample of the quantized latent. Due to the recursive nature of this process, obtaining the quantized latent ŷ is an autoregressive process.

[0117] According to this solution, the prediction of the second sample is performed using an autoregressive neural network or sub-network (prediction module). The prediction process may predict the mean value μ or the variance parameter σ.

[0118] According to a first alternative implementation of this solution, the quantized latent y^ samples are obtained by a process of entropy decoding. The mean value μ is obtained by a prediction module using the previously obtained y^ samples. The entropy coding module then obtains the next y^ samples using the mean value. The quantized residual samples can be obtained according to the following formula: w^=y^-μ After obtaining the samples of y^, we modify them to obtain the quantized latent samples y s ^ and obtain the quantized latent sample y s ̂ is then used by the synthesis transform to obtain the reconstructed image.

[0119] Alternative Implementation 2 Figure 13 shows another possible implementation of the NN-based image compression solution, depicted from the decoder's point of view.

[0120] A second alternative implementation of the solution (shown in FIG. 13) is the same as the first alternative implementation, except for how the quantized latent and the quantized residual latent are obtained. According to the second alternative implementation, the output of the entropy decoding module (ED) is a quantized residual latent ŵ that is used to obtain ŷ. ŷ is obtained by ŵ and a mean value μ, which is obtained by applying a prediction module to previously obtained samples of ŷ. Since the mean value is obtained using previously obtained samples, the prediction module is an autoregressive process. Furthermore, the prediction module includes a neural network-based network or sub-network.

[0121] After the samples of y^ are obtained, they are modified to obtain the quantized latent samples y s ^ and obtain the quantized latent sample y s ̂ is then used by the synthesis transform to obtain the reconstructed image.

[0122] Alternative Implementation 3 Another possible implementation of the solution in NN-based image compression is depicted in FIG. 14. It is depicted from the decoder's point of view. According to this alternative implementation, the quantized latent samples are multiplied by a multiplier after being used by the prediction module to obtain modified quantized latent samples. The modified quantized latent samples are used by the synthesis transform (denoted as "decoder") to obtain a reconstructed image. In all the alternative implementations mentioned above, the values ​​of the multipliers (scale1 and scale2), or a representation thereof, may be obtained from the bitstream. It should be noted that the term decoder can be used to refer to both the "synthetic transform" and the entire decoding process (including the synthetic transform, the prediction module, and the entropy decoding module).

[0123] 4.4.4. Further details of the solution According to this solution, the modified quantized latent y s ^ is given by one of the following formulas, where m1, m2 and m3 are multipliers and can be scalars of vectors. y s ^=y^xm1+μxm2+w^xm3 or y s ^=y^xm1+μxm2 or y s ^=y^xm1+w^xm2 or y s ^=y^+μxm1+w^xm2 or, y s ^=y^+μxm1 or y s ^=y^+w^xm1 or y s^=μxm1+w^xm2 or y s ^=y^+y^xm1 or y s ^=y^xm1 ... Furthermore, alternatively, the quantized potential y s ^ can be found according to one of the following formulas: where a1 is a sum term, which can be a scalar number or a vector of numbers. y s ^=y^+a1 The value of summation term a1 may be included in the bitstream by the encoder and decoded from the bitstream by the decoder.

[0124] The additive term a1 and the multiplicative terms m1, m2 and m3 can be applied together. For example: y s ^=y^xm1+μxm2+w^xm3+a1 or, y s ^=y^xm1+μxm2+a1 or y s ^=y^xm1+w^xm2+a1 or y s ^=y^+μxm1+w^xm2+a1 or y s ^=y^+μxm1+a1 or y s ^=y^+w^xm1+a1 or y s ^=μxm1+w^xm2+a1 or y s ^=y^+y^xm1+a1 or y s ^=y^xm1+a1 ... According to one exemplary implementation, the modified y s The samples of ̂ are obtained according to at least two of, but not limited to, the quantized latent (ŷ), the residual quantized latent (ŵ) and the mean (μ). For example, the modified quantized latent (y s^) can be obtained according to y^ and w^, or according to y^, w^ and μ. According to this solution, the modified quantized latent y s At least one sample of ̂ differs from the quantized potential ŷ.

[0125] The entropy decoder module may use the variance and mean parameters to obtain a probability distribution, such as a Gaussian distribution. For example, a Gaussian distribution has the formula: (outside 5) TIFF2025506992000010.tif28161, where the variance σ and mean μ parameters are used in this formula. The probability distribution is then used to obtain the quantized residual latent or quantized latent.

[0126] The corrected quantized latent y s ŷ is used to obtain a reconstructed image by applying a synthesis transformation. The synthesis transformation may be a neural network-based sub-network composed of convolutional or deconvolutional layers. This is the core of the solution, i.e., before the synthesis transformation, samples of the quantized latent ŷ are used to obtain further samples of ŷ. In other words, the core of the solution includes the following steps: 1. Get the first sample of the quantized latent ŷ. 2. Use the first sample to predict the second sample of ŷ. 3. Use the prediction to get the second sample of ŷ. 4. Correct the first sample and use the corrected quantized latent y s Get ^. 5. y s ^ is processed by the synthesis transform to obtain the reconstructed image. The value of the multiplier (or a representation thereof) may be included in the bitstream and decoded by a decoder. The samples of the quantized latent ŷ may be selectively modified according to rules. For example, the rules may include one of the following: - Samples of the quantized latent y^ are corrected when the value of the associated probability parameter is less than (or greater than) a threshold. For example, when the value of the dispersion parameter σ[c,i,j] is less than the threshold, the sample at the corresponding position (e.g., y^[:,i,j]) is corrected to obtain y s ^[:,i,j], while when σ[c,i,j] is greater than or equal to the threshold, y^[:,i,j] is set equal to obtain y s ^[:,i,j]. - To determine which samples of y^ to correct, a function of the probability parameter can be used. For example, the expression F(σ[c,i,j]) < thr can be used to determine whether to correct the corresponding sample at position [c,i,j]. o The probability parameter can be a dispersion value. o This function may be in the form of a probability distribution or a cumulative probability distribution. In one example, there is a Gaussian probability distribution (e.g., (Outer 6) TIFF2025506992000011.tif28161 or a Laplacian distribution, etc. - Samples of the quantized latent y^ are corrected according to their index values. For example, when the value of index c is equal to a predetermined index value K, the sample y^[c,i,j] may be corrected. Or, in another example, when index i is greater than a predetermined index value T, the sample y^[c,i,j] may be corrected. Index c may indicate the "channel number" of the "feature map ID". On the other hand, indices i,j indicate the spatial coordinates of the sample. The values of K and / or T may be included in the bitstream and obtained from the bitstream by the decoder. Alternatively, the values of K and / or T may be predetermined. A combination of checking whether the index is "greater than", "less than", "equal to" a value may be adopted to determine whether the sample y^[c,i,j] belongs to the first subset.

[0127] 4.4.5. Solution-related encoding process In the encoder, y s^ is obtained in the same way as the decoder. For example, y s ^ is calculated according to one of the following formulas: y s ^=y^xm1+μxm2+w^xm3 or y s ^=y^xm1+μxm2 or y s ^=y^xm1+w^xm2 or y s ^=y^+μxm1+w^xm2 or, y s ^=y^+μxm1 or y s ^=y^+w^xm1 or y s ^=μxm1+w^xm2 or y s ^=y^+y^xm1 or y s ^=y^xm1 or y s ^=y^xm1+μxm2+w^xm3+a1 or, y s ^=y^xm1+μxm2+a1 or y s ^=y^xm1+w^xm2+a1 or y s ^=y^+μxm1+w^xm2+a1 or y s ^=y^+μxm1+a1 or y s ^=y^+w^xm1+a1 or y s ^=μxm1+w^xm2+a1 or y s ^=y^+y^xm1+a1 or y s ^=y^xm1+a1 y s ^=y^+a1 ... The encoder determines the value of the multiplier according to a quality metric. In one example, different values ​​of the multiplier are tested to determine which multiplier results in the highest quality. The quality metric may be mean square error, or MS-SSIM, etc. The multiplier (or a representation thereof) may then be included in the bitstream so that a decoder can reproduce the results obtained by the encoder.

[0128] 4.5. Solution Advantages By modifying the quantized latent before performing the synthesis transform, artifacts caused by the transformation process can be at least partially removed, and thus the reconstructed image may be more visually pleasing.

[0129] 5. Embodiments 1. Decoder embodiment The method for decoding an image or video comprises at least one of the following steps: - Using the bitstream and a prediction module, obtain the first sample of the quantized latent ŷ. - Using said first sample to obtain a second sample of the quantized latent ŷ. - the corrected quantized latent y^ according to the first sample of the quantized latent y^ s Take the first sample of ^. The corrected quantized latent y s ^ to obtain the reconstructed image, and this corrected quantized latent y s ^ is used with composite transformations. 2. According to embodiment 1, The corrected quantized latent y s ^ Getting the first sample of - obtaining an additive term; - the first sample of the quantized latent y^ and the summation term according to the modified quantized latent y s and obtaining a first sample of ^. 3. According to embodiment 2, The step of obtaining the sum term includes obtaining a first sample of the quantized residual ŵ from the bitstream. 4. According to embodiment 3, Said first sample of the quantized residual ŵ is used to obtain a first sample of the quantized latent ŷ. 5. According to embodiment 2, The step of obtaining the sum term includes obtaining a first sample of the mean μ using a prediction module. 6. According to embodiment 5, Said first sample of the mean μ is used to obtain a first sample of the quantized potential ŷ. 7. According to embodiments 5 and 6, The first sample of the mean μ is obtained by the prediction module using at least one other sample of the quantized potential ŷ. 8. According to embodiment 2, The step of obtaining the sum term includes obtaining a first sample of the mean μ from the bitstream. 9. According to embodiments 5 to 8, Said first sample of the mean μ is used to obtain a first sample of the quantized potential ŷ. 10. According to embodiment 2, The value of the sum term (or a representation thereof) is taken from the bitstream. 11. According to embodiment 2, The summation term is the quantized latent y^, and the modified quantized latent y s ^ is y s ^=y^xm1 or y s ^=y^+y^xm2, where m1 is not equal to 1 and m2 is not equal to 0. 12. According to embodiment 1, The first sample of the modified quantized latent is y s It is obtained according to ^=y^xm1. 13. According to any of the above embodiments, The step of obtaining the sum term involves subtracting samples of the mean μ from samples of the quantized latent ŷ. 14. According to any of the above embodiments, The step of obtaining the sum term includes subtracting the first sample of the mean μ from the first sample of the quantized latency ŷ. 15. According to any of the above embodiments, The quantized potential ŷ and the modified quantized potential yŝ include at least a second sample, where the first sample of the quantized potential is used by the prediction module to obtain a second sample of the mean μ. 16. According to embodiment 15, The second sample of the mean μ is used to obtain a second sample of the quantized latent ŷ. 17. According to embodiment 16, The corrected quantized latent y s The second sample of ŷ is obtained by using the second sample of the quantized latent ŷ. 18. According to any of the above embodiments, The prediction module includes a neural sub-network. 19. According to any of the above embodiments, The prediction module is an autoregressive sub-network. 20. According to any of the above embodiments, The prediction module receives as input the previously obtained quantized latent ŷ and processes the input. 21. According to any of the above embodiments, The prediction module includes a context sub-network or a hyper-decoder sub-network. 22. According to any of the above embodiments, The multiplier (or a representation of it) is included in the bitstream. 23. According to any of the above embodiments, The multiplier is a scalar value different from 0. 24. According to any of the above embodiments, The multiplier is a vector. 25. According to any of the above embodiments, The synthesis transform is a neural sub-network. 26. According to any of the above embodiments, the method includes at least one control information value capable of determining whether and / or how any of the above methods are applied. 27. Whether and / or how to apply any of the above methods may depend on conditions such as color format and / or color components. 28. According to any of the above embodiments, any value included in the bitstream may be coded at the sequence / picture / slice / block level. 29. According to any of the above embodiments, any value contained in the bitstream may be binarized before being encoded. 30. According to any of the above embodiments, any value contained in the bitstream is coded with at least one arithmetic coding context.

[0130] In the following, details of embodiments of the present disclosure relating to neural network-based visual data encoding are described. As used herein, the term "visual data" may refer to images, pictures in a video, or any other visual data suitable for being encoded.

[0131] As mentioned above, existing designs directly use the quantized latent representation of visual data in a synthesis transformation to reconstruct the visual data. However, traditional transformation processes can introduce artifacts that make the reconstructed visual data visually unattractive.

[0132] To solve the above problems and some other problems not mentioned, the visual data processing solution described below is disclosed.

[0133] 15 illustrates an example visual data decoding process 1500 according to some embodiments of the present disclosure. For example, the visual data decoding process 1500 may be performed by the visual data decoder 124 shown in FIG. It should be understood that the visual data decoding process 1500 may also include additional blocks not shown and / or blocks that are shown may be omitted. The scope of the present disclosure is not limited in this respect.

[0134] As shown in FIG. 15, the bitstream may be input to an entropy decoder 1520. In the entropy decoder 1520, an entropy decoding process may be performed on the bitstream to obtain a quantized residual latent representation of the visual data (denoted as ŵ in FIG. 15). Note that the quantized residual latent representation may also be referred to as a quantization of the residual latent representation. Similarly, the quantized latent representation may also be referred to as a quantization of the latent representation. The residual latent representation and / or the quantized residual latent representation may indicate a difference between the latent representation of the visual data and a prediction of the latent representation.

[0135] As a non-limiting example, the entropy decoding process may be performed based on the probability distribution information generated by a factorization entropy sub-network (not shown in FIG. 15). In some embodiments, the factorization entropy sub-network may generate the probability distribution information by using a predefined template, e.g., a predefined mean and variance in the case of a Gaussian distribution. The entropy decoding process performed by the entropy decoder 1520 may be an arithmetic decoding process, a Huffman decoding process, etc.

[0136] In block 1518, a quantized latent representation of the visual data (denoted as ŷ in FIG. 15) may be obtained by adding the quantized residual latent representation and a prediction of the latent representation (denoted as μ in FIG. 15) generated by a prediction model 1522. The prediction model 1522 may be implemented with a neural network. Note that in this application, a model may also be referred to as a module. That is, a prediction model may also be referred to as a prediction module, and an estimation model may also be referred to as an estimation module. In some embodiments, this process may be determined at the sample level. In other words, a sample of the quantized latent representation may be obtained by adding a corresponding sample of the quantized residual latent representation and a prediction of the sample.

[0137] In some embodiments, the prediction model 1522 may be autoregressive. In one example, the prediction model 1522 may include a context sub-network and a prediction fusion sub-network (not shown in FIG. 15). At least one reconstructed sample of the quantized latent representation ŷ may be input to the context sub-network. The at least one reconstructed sample was previously obtained and is used to generate a prediction of the current sample of the quantized latent representation ŷ without adjustment. The context sub-network may generate intermediate information based on the at least one reconstructed sample. As a non-limiting example, the intermediate information may reflect an average value of the at least one reconstructed sample. The prediction fusion sub-network may generate a prediction of the current sample based on the intermediate information. In one example, the prediction may indicate a predicted average value of the current sample. The obtained prediction is summed with a corresponding sample of the quantized residual latent representation ŵ to reconstruct the current sample. The above process may be repeated to reconstruct all samples of the quantized latent representation ŷ.

[0138] It should be appreciated that the context sub-network 1426 may also be referred to as a context model, a context model sub-network, and / or the like. Additionally, the predictive fusion sub-network may also be referred to as a fusion sub-network, a predictive sub-network, and / or the like. Additionally, the predictive model 1522 may also be implemented in any other suitable manner, for example, a multi-stage context model may be employed, or the predictive fusion sub-network may be eliminated. The scope of the present disclosure is not limited in this respect.

[0139] In block 1514, the quantized latent representation ŷ is converted into intermediate information of the visual data (y s The intermediate representation may be updated to obtain a quantized latent representation, denoted as ŷ. The intermediate representation is different from the quantized latent representation. In some embodiments, the update process may be performed at the sample level. In other words, the update process is performed sample by sample. Alternatively, after all samples of the quantized latent representation ŷ are obtained, the update process may be performed on the entire quantized latent representation.

[0140] In some embodiments, at least a portion of the quantized latent representation may be updated based on at least one of the at least one parameter, a prediction of at least a portion of the quantized latent representation, or a difference between the prediction and the quantized latent representation. As non-limiting examples, the at least one parameter may be determined based on an encoder-side or decoder-side quality metric. The quality metric may include mean square error, structural similarity (SSIM), multi-scale structural similarity (MS-SSIM), and / or the like. In one example, the at least one parameter may be a scalar value or vector different from zero. The at least one parameter may be signaled in the bitstream. Alternatively, an indication of the at least one parameter may be signaled in the bitstream.

[0141] In some embodiments, at least some of the quantized latent representations may be updated by scaling at least some of them with a first parameter. Additionally or alternatively, at least some of the quantized latent representations may be updated based on a product of a prediction and a second parameter and / or a product of a difference and a third parameter. In some additional or alternative embodiments, at least some of the quantized latent representations may be updated by adding a fourth parameter, as described in detail above in Section 4.4.4, where equations that may be used for the update process are listed for illustrative and non-limiting purposes.

[0142] In some embodiments, different samples of the quantized latent representation may be adjusted in different manners, such as with different parameters. Additionally or alternatively, a set of samples of the quantized latent representation may not be updated. In other words, samples of the quantized latent representation may be selectively updated.

[0143] In some embodiments, a grouping process may be performed in the update block for selective updating 1514 to determine at least a portion of the quantized latent representation that needs to be updated or updated in a particular manner. In one example, the grouping process may be performed at a sample level. In other words, the at least a portion decision is made for each sample of the quantized latent representation. Alternatively, the grouping process may be performed at a block level. More specifically, the samples of the quantized residual latent representation may be divided into a number of blocks. Each of the number of blocks may have a predetermined size of N×M, such as 8×8. Each of N and M is a positive integer. N and M may be indicated in the bitstream. The samples may be grouped based on the number of blocks. In other words, the at least a portion decision is made for each of the number of blocks, and samples within the same block are divided into the same set.

[0144] In some embodiments, for a sample of the quantized latent representation, if a statistical value corresponding to the sample is less than a first threshold, the at least portion may be determined to include the sample, and the sample may be grouped into a first set. Otherwise, the sample may be determined not to be included in the at least portion, and the sample may be grouped into a second set different from the first set. As a non-limiting example, the statistical value may be a variance, such as a variance of a Gaussian probability. The variance may be generated based on the bitstream using a hyper-scale decoder sub-network. It should be appreciated that any other suitable statistical value, such as a mean or standard deviation, may also be used. Additionally, the statistical value may be referred to as a statistical parameter, a probability parameter, a probability distribution parameter, or the like. The scope of the present disclosure is not limited in this respect.

[0145] Alternatively, for a sample of the quantized latent representation, if a statistical value corresponding to the sample is greater than another threshold, which may be the same as or different from the first threshold, the at least portion may be determined to include the sample, and the sample may be grouped into the first set. Otherwise, the sample may be determined not to be included in the at least portion, and the sample may be grouped into the second set. The above comparison may be performed for each of the samples of the quantized residual latent representation to obtain the at least portion.

[0146] It should be appreciated that the at least some may be determined in any other suitable manner, for example, based on a comparison between a threshold and a value determined based on a statistical value, or based on a comparison between a threshold and an index of the sample, for example, the index of the sample may indicate a channel number of the sample, a feature map identifier of the sample, or a spatial coordinate of the sample, as described in more detail above in section 4.4.4.

[0147] In some alternative embodiments, the grouping process may be performed at the block level. In such a case, for samples in a block of the quantized latent representation, if a statistical value corresponding to each sample in the block is less than a fourth threshold, it may be determined that the at least one portion includes all samples in the block, and these samples may be grouped into a first set. Otherwise, it may be determined that the samples are not included in the at least one portion, and these samples may be grouped into a second set different from the first set.

[0148] Alternatively, for samples in a block of the quantized latent representation, if a statistical value corresponding to each sample in the block is greater than another threshold, which may be the same as or different from the fourth threshold, it may be determined that the at least portion includes all samples in the block and these samples may be grouped into a first set. Otherwise, it may be determined that the samples are not included in the at least portion and these samples may be grouped into a second set. The above comparison may be performed for each of the blocks of the quantized residual latent representation to obtain the at least portion.

[0149] It should be understood that the thresholds may be determined at least in part in any other suitable manner, for example, based on a comparison of a threshold to a metric determined based on a statistical value corresponding to the sample, or based on a comparison of a threshold to an index of each of the samples. By way of example, the metric may be an average, a minimum, a maximum, a particular function, etc. In one example, at least one of the thresholds may be indicated in the bitstream. Alternatively, at least one indication of the at least one threshold may be indicated in the bitstream.

[0150] Alternatively, it may be determined on a sample-by-sample or block-by-block basis whether a sample or samples in a block need to be updated, and after the determination, an update process may be performed on the corresponding sample or samples in the block. As a non-limiting example, for a sample of the quantized latent representation, it may be determined that the sample is to be updated if a statistical value corresponding to the sample is less than a threshold. The sample may then be updated based on the update process described above. Similarly, for samples in a block of the quantized latent representation, it may be determined that the samples in the block are to be updated if a statistical value corresponding to each sample in the block is less than a threshold. The samples may then be updated based on the update process described above. It should also be noted that this determination may be made based on any other suitable parameter(s) and / or value(s), such as a value determined based on the statistical value or an index of the sample. The scope of the present disclosure is not limited in this respect.

[0151] After obtaining at least a portion of the quantized latent representation, the above-mentioned update process may be performed thereon. It should be understood that the at least a portion of the quantized latent representation may include one or more samples, or even all samples, of the quantized latent representation. When the at least a portion of the quantized latent representation includes all samples of the quantized latent representation, the intermediate representation corresponds to the result of updating at least a portion.

[0152] In the synthesis transformation block 1512, the intermediate representation y is transformed to obtain the reconstructed visual data 1510. s A synthetic transformation may be performed on ^. For example, the synthetic transformation may be performed using a neural network-based sub-network. It should be noted that at least some of the samples used for the synthetic transformation are different from the samples used for the predictive model 1522.

[0153] From the above, an intermediate representation different from the quantized latent representation of the visual data is generated and used for synthesis transformation. Compared with the conventional solution in which the quantized latent representation is directly used for synthesis transformation, the proposed method can at least partially remove the artifacts caused by the conventional transformation process, and thus the reconstructed image can be more visually pleasing. This allows the proposed method to advantageously improve the coding quality.

[0154] 16 illustrates another exemplary visual data decoding process 1600 according to some embodiments of the present disclosure. For example, the visual data decoding process 1600 may be performed by the visual data decoder 124 illustrated in FIG. It should be understood that the visual data decoding process 1600 may also include additional blocks not illustrated and / or illustrated blocks may be omitted. The scope of the present disclosure is not limited in this respect.

[0155] As shown in FIG. 16, the bitstream may be input to an entropy decoder 1620. In the entropy decoder 1620, an entropy decoding process is performed on the bitstream to obtain a quantized residual latent representation of the visual data (denoted as ŵ in FIG. 16). The entropy decoding process may be performed based on a first statistic generated by an estimation model 1622. The estimation model 1622 may also be implemented with a neural network. As a non-limiting example, the first statistic may be a variance (denoted as variance σ in FIG. 16), such as the variance of a Gaussian probability. It should be appreciated that any other suitable statistic may also be used, such as the mean or standard deviation. The scope of the present disclosure is not limited in this respect.

[0156] In addition to the first statistical value, the estimation model 1622 further generates a second statistical value (denoted as μ in FIG. 16 ). In some embodiments, the second statistical value may indicate a prediction of the latent representation of the visual data. As a non-limiting example, the second statistical value may be an average, such as an average of Gaussian probabilities. In one example, the first and second statistical values ​​may be jointly generated by the estimation model 1622. Alternatively, the estimation model 1622 may include a first sub-network for generating the first statistical value and a second sub-network for generating the second statistical value. As a non-limiting example, the first sub-network may be a hyperscale decoder sub-network and the second sub-network is a hyperscale decoder sub-network.

[0157] In the generation block 1614, an intermediate representation of the visual data (y s The intermediate representation y(x,y) may be generated based on the second statistic μ and the quantized residual latent representation ŵ. Note that the second statistic μ indicates a prediction of the latent representation and may hereinafter also be referred to as a “prediction”. Similarly, the quantized residual latent representation ŵ indicates a difference between the prediction and the latent representation and may hereinafter also be referred to as a “difference”. In some embodiments, at least a portion of the intermediate representation may be generated by adding a product of the prediction and the first parameter and a product of the difference and the second parameter. Alternatively, at least a portion of the intermediate representation may be generated by adding a product of the prediction and the first parameter, a product of the difference and the second parameter, and a third parameter. This is described in more detail in section 4.4.4 above, and may be used to generate an intermediate representation y(x,y) based on the prediction μ and the quantized residual latent representation ŵ. s The formulas that can be used to generate ^ are listed for illustrative and non-limiting purposes.

[0158] Note that the sum of the prediction μ and the quantized residual latent representation ŵ is the quantized latent representation ŷ. Therefore, the generation process above can be considered equivalent to the update process described with respect to update block 1514 in FIG. 15. In this way, the intermediate representation ŷ s^ may also be selectively generated, similar to the selective updating described above.

[0159] In the synthesis transformation block 1612, the intermediate representation y is transformed to obtain the reconstructed visual data 1610. s A synthesis transformation may be performed on ^. For example, the synthesis transformation may be performed by using a neural network-based sub-network.

[0160] From the above, an intermediate representation different from the quantized latent representation of the visual data is generated and used for synthesis transformation. Compared with the conventional solution in which the quantized latent representation is directly used for synthesis transformation, the proposed method can at least partially remove the artifacts caused by the conventional transformation process, and thus the reconstructed image can be more visually pleasing. This allows the proposed method to advantageously improve the coding quality.

[0161] Although two exemplary visual data decoding processes have been described above with respect to Figures 15 and 16, it should be understood that any other suitable variations of the visual data decoding processes are also contemplated in view of the present disclosure, such as two further exemplary visual data decoding processes illustrated by way of example in Figures 12 and 14.

[0162] Furthermore, although the proposed method is described from the perspective of a decoder, it may also be implemented in an encoder, such as for the purpose of reconstructing encoded visual data. The scope of the present disclosure is not limited in this respect. It should be understood that the above illustrations and / or examples are provided for illustrative purposes only. The scope of the present disclosure is not limited in this respect. The embodiments of the present disclosure should be considered as examples for illustrating the general concept and should not be construed in a narrow sense. Furthermore, these embodiments may be applied individually or combined in any manner.

[0163] FIG. 17 illustrates a flowchart of a method 1700 for visual data processing according to some embodiments of the present disclosure. The method 1700 may be performed during conversion between visual data and a bitstream of visual data. As shown in FIG. 17, the method 1700 begins with obtaining 1702 an intermediate representation of the visual data. The intermediate representation is different from a quantized latent representation of the visual data. Furthermore, the intermediate representation is generated based on at least one of at least one parameter, at least a portion of the quantized latent representation, a prediction of at least a portion of the quantized latent representation, or a difference between the prediction and at least a portion of the quantized latent representation. The quantized latent representation is generated based on applying a first neural network to the visual data.

[0164] In some embodiments, an analytical transformation may be performed on the visual data by using a first neural network to obtain a latent representation of the visual data. The visual data may include one or more pictures in an image or video. A first statistical value may be subtracted from the latent representation to obtain a residual latent representation. The first statistical value is generated by a second neural network and indicates a prediction of the latent representation. As a non-limiting example, the first statistical value may be the mean of a probability distribution, such as a Gaussian probability distribution. For example, the probability distribution may describe a probability distribution of values ​​of one or more samples of the latent representation.

[0165] At 1704, a composition transform is performed on the intermediate representation for transformation. In one example, the transformation may include encoding the visual data into a bitstream. Alternatively or additionally, the transformation may include decoding the visual data from the bitstream. It should be appreciated that the above illustrations are provided for illustrative purposes only. The scope of the present disclosure is not limited in this respect.

[0166] From the above, an intermediate representation different from the quantized latent representation of the visual data is generated and used for synthesis transformation. Compared with the conventional solution in which the quantized latent representation is directly used for synthesis transformation, the proposed method can at least partially remove the artifacts caused by the conventional transformation process, and thus the reconstructed image can be more visually pleasing. This allows the proposed method to advantageously improve the coding quality.

[0167] In some embodiments, at least a portion of a quantized latent representation may be obtained at 1702. Further, at least a portion of the quantized latent representation may be generated based on at least one of the at least one parameter, the prediction, or the differential to obtain at least a portion of an intermediate representation.

[0168] In some embodiments, at least a portion of the quantized latent representation may be updated by scaling at least a portion of the quantized latent representation with a first parameter of the at least one parameter. Additionally or alternatively, at least a portion of the quantized latent representation may be updated based on a product of the prediction and a second parameter of the at least one parameter and / or a product of the difference and a third parameter of the at least one parameter. Additionally or alternatively, at least a portion of the quantized latent representation may be updated by adding a fourth parameter of the at least one parameter.

[0169] In some embodiments, the prediction may be a mean. The difference may be included in a quantized residual latent representation of the visual data. In some embodiments, an entropy decoding process may be performed on the bitstream to obtain at least a portion of the quantized latent representation. In some embodiments, the method may further include generating the prediction by using the first model. In some embodiments, the difference may be obtained by performing an entropy decoding process on the bitstream. The prediction may be generated by using the first model. At least a portion of the quantized latent representation may be generated based on the prediction and the difference.

[0170] In some embodiments, predictions of samples in at least a portion of the quantized latent representation may be generated based on at least one reconstructed sample of the quantized latent representation by using a first model. In one example, the first model may be a predictive model. As a non-limiting example, the first model may be an autoregressive one. For example, the first model may include a context sub-network or a context model sub-network.

[0171] In some embodiments, at least a portion of the quantized latent representation may include all samples of the quantized latent representation, and the intermediate representation corresponds to a result of the updating. In some further embodiments, the method may further include updating a further portion of the quantized latent representation based on at least one further parameter that is different from the at least one parameter, the further portion being different from the at least a portion of the quantized latent representation.

[0172] In some embodiments, at 1702, an intermediate representation may be generated based at least in part on the prediction and the difference.

[0173] In some embodiments, at least a portion of the intermediate representation may be generated by adding a product of the prediction and a first parameter of the at least one parameter and a product of the difference and a second parameter of the at least one parameter. Alternatively, at least a portion of the intermediate representation may be generated by adding a product of the prediction and the first parameter, a product of the difference and the second parameter, and a third parameter of the at least one parameter.

[0174] In some embodiments, at least one of the prediction or the difference may be generated by using a first model. In one example, the first model may be an estimation model. For example, the first model may include a neural network-based sub-network. The input of the first model may include a bitstream. In some embodiments, the first model may include a first sub-network for generating the prediction and a second sub-network for generating the statistics. As a non-limiting example, the first sub-network may be a hyperdecoder sub-network and the second sub-network may be a hyperscale decoder sub-network.

[0175] In some embodiments, the method may further include determining at least a portion of the quantized latent representation from the quantized latent representation. In one example, whether the at least a portion of the quantized latent representation includes a sample of the quantized latent representation may be determined based on at least one of a comparison of a first threshold to a statistical value corresponding to the sample, a comparison of a second threshold to a value determined based on the statistical value, or a comparison of a third threshold to an index of the sample.

[0176] Alternatively, whether at least a portion of the quantized latent representation includes a sample in a block of the quantized latent representation may be determined based on at least one of a comparison of a statistical value corresponding to each of the samples to a fourth threshold, a comparison of a metric determined based on the statistical value corresponding to the samples to a fifth threshold, or a comparison of an index of each of the samples to a sixth threshold. In some embodiments, the metric may be an average, minimum, or maximum value. The index of the sample may indicate one of a channel number of the sample, a feature map identifier of the sample, or a spatial coordinate of the sample.

[0177] In some embodiments, at least one of the following thresholds or an indication of at least one of the following thresholds may be indicated in the bitstream: a first threshold, a second threshold, a third threshold, a fourth threshold, a fifth threshold, or a sixth threshold. In some embodiments, the statistic may be a variance. For example, the statistic may be a variance of Gaussian probabilities. The statistic may be obtained based on the bitstream by using a second sub-network.

[0178] In some embodiments, at least one parameter or an indication of the at least one parameter may be included in the bitstream. For example, the at least one parameter may be a scalar value or a vector different from zero. In one example, the at least one parameter may be determined based on a quality metric. As non-limiting examples, the quality metric may include at least one of mean square error, structural similarity (SSIM), or multi-scale structural similarity (MS-SSIM).

[0179] In some embodiments, at least a portion of the quantized latent representation may include one or more samples of the quantized latent representation. In some embodiments, the synthesis transformation may be performed by using a neural network-based sub-network.

[0180] In some embodiments, at least one of information regarding whether to apply the method or information regarding how to apply the method may be indicated in the bitstream, hi some alternative embodiments, at least one of information regarding whether to apply the method or information regarding how to apply the method may depend on the color format and / or color components of the visual data.

[0181] In some embodiments, the values ​​included in the bitstream may be coded at one of a sequence level, a picture level, a slice level, or a block level. In some embodiments, the values ​​included in the bitstream may be binarized before being coded. In some embodiments, the values ​​included in the bitstream are coded with at least one arithmetic coding context.

[0182] According to a further embodiment of the present disclosure, there is provided a non-transitory computer-readable recording medium storing a bitstream of visual data generated by a method performed by an apparatus for visual data processing, the method including obtaining an intermediate representation of the visual data, the intermediate representation being different from a quantized latent representation of the visual data and generated based on at least one of at least one parameter, at least a portion of the quantized latent representation, a prediction of at least a portion of the quantized latent representation, or a difference between the prediction and at least a portion of the quantized latent representation, and generating the bitstream based on a synthesis transformation on the intermediate representation, the quantized latent representation being generated based on applying a first neural network to the visual data.

[0183] According to yet another embodiment of the present disclosure, there is provided a method for storing a bitstream of visual data, the method including obtaining an intermediate representation of the visual data, the intermediate representation different from a quantized latent representation of the visual data and generated based on at least one of at least one parameter, at least a portion of the quantized latent representation, a prediction of at least a portion of the quantized latent representation, or a difference between the prediction and at least a portion of the quantized latent representation, generating a bitstream based on a synthesis transformation on the intermediate representation, and storing the bitstream on a non-transitory computer-readable storage medium, the quantized latent representation generated based on applying a first neural network to the visual data.

[0184] Implementations of the present disclosure can be described in terms of the following clauses, the features of which can be combined in any reasonable manner.

[0185] Clause 1. A method for visual data processing, comprising: obtaining an intermediate representation of the visual data for conversion between visual data and a bitstream of the visual data, the intermediate representation being different from a quantized latent representation of the visual data and being generated based on at least one of at least one parameter, at least a portion of the quantized latent representation, a prediction of the at least a portion of the quantized latent representation, or a difference between the prediction and the at least a portion of the quantized latent representation; and performing a synthesis transform on the intermediate representation for conversion, the quantized latent representation being generated based on applying a first neural network to the visual data.

[0186] Clause 2. The method of clause 1, wherein obtaining the intermediate representation includes obtaining at least a portion of the quantized latent representation; and updating the at least a portion of the quantized latent representation based on at least one of the at least one parameter, the prediction, or the difference, to obtain at least a portion of the intermediate representation.

[0187] Clause 3. The method of clause 2, wherein updating the at least a portion of the quantized latent representation includes at least one of: scaling the at least a portion of the quantized latent representation by a first parameter of the at least one parameter; updating the at least a portion of the quantized latent representation based on a product of the prediction and a second parameter of the at least one parameter and / or a product of the difference and a third parameter of the at least one parameter; or updating the at least a portion of the quantized latent representation by adding at least a fourth parameter of the at least one parameter.

[0188] Clause 4. The method of clause 2 or 3, wherein the prediction is a mean or the difference is included in a quantized residual latent representation of the visual data.

[0189] Clause 5. The method according to any one of clauses 2 to 4, wherein obtaining at least a portion of the quantized latent representation includes performing an entropy decoding process on the bitstream to obtain at least a portion of the quantized latent representation.

[0190] Clause 6. The method of clause 5, further comprising generating the prediction by using a first model.

[0191] Clause 7. The method of any one of clauses 2 to 4, wherein obtaining at least a portion of the quantized latent representation includes obtaining the difference by performing an entropy decoding process on the bitstream, generating the prediction by using a first model, and generating the at least a portion of the quantized latent representation based on the prediction and the difference.

[0192] Clause 8. The method of clause 6 or 7, wherein generating the prediction includes generating a prediction of a sample within the at least a portion of the quantized latent representation based on at least one reconstructed sample of the quantized latent representation by using the first model.

[0193] Clause 9. The method of any one of clauses 6 to 8, wherein the first model is a predictive model.

[0194] Clause 10. The method of any one of clauses 6 to 9, wherein the first model is autoregressive.

[0195] Clause 11. The method according to any one of clauses 6 to 10, wherein the first model includes a context sub-network or a context model sub-network.

[0196] Clause 12. A method according to any one of clauses 2 to 11, wherein the at least a portion of the quantized latent representation includes all samples of the quantized latent representation, and the intermediate representation corresponds to a result of the update.

[0197] Clause 13. The method of any one of clauses 2 to 11, further comprising updating a further portion of the quantized latent representation based on at least one further parameter different from the at least one parameter, the further portion being different from the at least a portion of the quantized latent representation.

[0198] Clause 14. The method of clause 1, wherein obtaining the intermediate representation includes generating at least a portion of the intermediate representation based on the prediction and the difference.

[0199] Clause 15. The method of clause 14, wherein generating at least a portion of the intermediate representation comprises generating at least a portion of the intermediate representation by adding a product of the prediction and a first parameter of the at least one parameter and a product of the difference and a second parameter of the at least one parameter, or generating at least a portion of the intermediate representation by adding a product of the prediction and the first parameter, a product of the difference and the second parameter, and a third parameter of the at least one parameter.

[0200] Clause 16. The method of clause 14 or 15, wherein at least one of the prediction or the difference is generated by using a first model.

[0201] Clause 17. The method of any one of clauses 6 to 8 and clause 16, wherein the first model is an estimated model.

[0202] Clause 18. A method according to any one of clauses 6 to 8 and clauses 16 to 17, wherein the first model comprises a neural network-based sub-network or the input of the first model comprises the bit stream.

[0203] Clause 19. A method according to any one of clauses 6 to 8 and clauses 16 to 17, wherein the first model includes a first sub-network for generating the prediction and a second sub-network for generating statistics.

[0204] Clause 20. The method of clause 19, wherein the first sub-network is a hyperdecoder sub-network and the second sub-network is a hyperscale decoder sub-network.

[0205] Clause 21. The method of any one of clauses 1 to 20, further comprising determining the at least a portion of the quantized latent representation from the quantized latent representation.

[0206] Clause 22. The method of clause 21, wherein determining the at least a portion of the quantized latent representation from the quantized latent representation includes determining whether the at least a portion of the quantized latent representation includes a sample of the quantized latent representation based on at least one of the following: a comparison of a first threshold value to a statistical value corresponding to the sample, a comparison of a second threshold value to a value determined based on the statistical value, or a comparison of an index of the sample to a third threshold value.

[0207] Clause 23. The method of clause 21, wherein determining the at least a portion of the quantized latent representation from the quantized latent representation includes determining whether the at least a portion of the quantized latent representation includes a sample within a block of the quantized latent representation based on at least one of the following: a comparison of a statistical value corresponding to each of the samples to a fourth threshold, a comparison of a metric determined based on a statistical value corresponding to the samples to a fifth threshold, or a comparison of an index of each of the samples to a sixth threshold.

[0208] Clause 24. The method of clause 23, wherein the metric is an average value, a minimum value, or a maximum value.

[0209] Clause 25. A method according to any one of clauses 22 to 24, wherein the index of a sample indicates one of a channel number of the sample, a feature map identifier of the sample, or a spatial coordinate of the sample.

[0210] Clause 26. A method according to any one of clauses 22 to 25, wherein at least one or at least one indication of the first threshold, the second threshold, the third threshold, the fourth threshold, the fifth threshold, or the sixth threshold is indicated in the bitstream.

[0211] Clause 27. The method of any one of clauses 19 to 26, wherein the statistical value is a variance.

[0212] Clause 28. A method according to any one of clauses 19 to 27, wherein the statistical value is a variance of a Gaussian probability or the statistical value is obtained based on the bit stream by using the second sub-network.

[0213] Clause 29. A method according to any one of clauses 1 to 28, wherein the at least one parameter or an indication of the at least one parameter is included in the bitstream.

[0214] Clause 30. The method of any one of clauses 1 to 29, wherein the at least one parameter is a scalar value or a vector different from zero.

[0215] Clause 31. The method of any one of clauses 1 to 30, wherein the at least one parameter is determined based on a quality metric.

[0216] Clause 32. The method of clause 31, wherein the quality metric includes at least one of mean square error, structural similarity (SSIM), or multi-scale structural similarity (MS-SSIM).

[0217] Clause 33. The method of any one of clauses 1 to 32, wherein the at least a portion of the quantized latent representation comprises one or more samples of the quantized latent representation.

[0218] Clause 34. A method according to any one of clauses 1 to 33, wherein the synthesis transformation is performed by using a neural network based sub-network or the first neural network is used to perform an analytical transformation on the visual data.

[0219] Clause 35. A method according to any one of clauses 1 to 34, wherein at least one of information regarding whether to apply the method or information regarding how to apply the method is indicated in the bitstream.

[0220] Clause 36. A method according to any one of clauses 1 to 34, wherein at least one of the information regarding whether to apply the method or the information regarding how to apply the method depends on the color format and / or color components of the visual data.

[0221] Clause 37. A method according to any one of clauses 1 to 36, wherein the values ​​included in the bitstream are coded at one of a sequence level, a picture level, a slice level or a block level.

[0222] Clause 38. A method according to any one of clauses 1 to 37, wherein the values ​​contained in the bitstream are binarised before being encoded.

[0223] Clause 39. A method according to any one of clauses 1 to 38, wherein values ​​included in the bitstream are coded with at least one arithmetic coding context.

[0224] Clause 40. The method of any one of clauses 1 to 39, wherein the visual data includes an image or a video picture.

[0225] Clause 41. The method of any one of clauses 1 to 40, wherein the transforming includes encoding the visual data into the bitstream.

[0226] Clause 42. The method of any one of clauses 1 to 40, wherein the converting includes decoding the visual data from the bitstream.

[0227] Clause 43. An apparatus for processing visual data comprising a processor and a non-transitory memory having instructions thereon, the instructions, when executed by the processor, causing the processor to perform a method according to any one of clauses 1 to 42.

[0228] Clause 44. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method of any one of clauses 1 to 42.

[0229] Clause 45. A non-transitory computer-readable recording medium storing a bitstream of visual data generated by a method performed by an apparatus for visual data processing, the method including: obtaining an intermediate representation of the visual data, the intermediate representation being different from a quantized latent representation of the visual data and being generated based on at least one of at least one parameter, at least a portion of the quantized latent representation, a prediction of the at least a portion of the quantized latent representation, or a difference between the prediction and the at least a portion of the quantized latent representation; and generating the bitstream based on a synthesis transform on the intermediate representation, the quantized latent representation being generated based on applying a first neural network to the visual data.

[0230] Clause 46. A method for storing a bitstream of visual data, comprising: obtaining an intermediate representation of the visual data, the intermediate representation being different from a quantized latent representation of the visual data and being generated based on at least one of at least one parameter, at least a portion of the quantized latent representation, a prediction of the at least a portion of the quantized latent representation, or a difference between the prediction and the at least a portion of the quantized latent representation; generating the bitstream based on a synthesis transformation on the intermediate representation; and storing the bitstream on a non-transitory computer-readable recording medium, wherein the quantized latent representation is generated based on applying a first neural network to the visual data.

[0231] Exemplary Devices 18 illustrates a block diagram of a computing device 1800 in which various embodiments of the disclosure may be implemented. The computing device 1800 may be implemented as or included in the source device 110 (or the visual data encoder 114) or the destination device 120 (or the visual data decoder 124).

[0232] It should be understood that the computing device 1800 shown in FIG. 18 is for illustrative purposes only and is not intended to suggest any limitation on the functionality and scope of the embodiments of the present disclosure.

[0233] 18, the computing device 1800 includes a general-purpose computing device 1800. The computing device 1800 may include at least one or more processors or processing units 1810, a memory 1820, a storage unit 1830, one or more communication units 1840, one or more input devices 1850, and one or more output devices 1860.

[0234] In some embodiments, the computing device 1800 may be implemented as any user terminal or server terminal having computing capabilities. The server terminal may be a server provided by a service provider, a large-scale computing device, etc. The user terminal may be, for example, any type of mobile, fixed, or portable terminal, including a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device, or any combination thereof, further including accessories and peripherals of these devices, or any combination thereof. It is contemplated that the computing device 1800 may support any type of interface to a user, such as "wearable" circuitry.

[0235] The processing unit 1810 may be a physical or virtual processor and may perform various processes based on programs stored in the memory 1820. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of the computing device 1800. The processing unit 1810 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0236] Computing device 1800 typically includes a variety of computer storage media. Such media may be any media accessible by computing device 1800, including but not limited to volatile and non-volatile media, or removable and non-removable media. Memory 1820 may be volatile memory (e.g., registers, cache, Random Access Memory (RAM)), non-volatile memory (Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), or flash memory, etc.), or any combination thereof. Storage unit 1830 may be any removable or non-removable media and may include machine-readable media such as memory, flash memory drives, magnetic disks, or other media that may be used to store information and / or visual data and that are accessible within computing device 1800.

[0237] Computing device 1800 may further include other removable / non-removable, volatile / non-volatile memory media. Although not shown in FIG. 18, it is possible to provide a magnetic disk drive for reading from and / or writing to a removable, non-volatile magnetic disk, and an optical disk drive for reading from and / or writing to a removable, non-volatile optical disk. In such a case, each drive may be connected to a bus (not shown) via one or more optical data media interfaces.

[0238] The communication unit 1840 communicates with further computing devices via a communication medium. Furthermore, the functionality of the components in the computing device 1800 may be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the computing device 1800 may operate in a networked environment using logical connections with one or more other servers, networked personal computers (PCs), or even more general network nodes.

[0239] The input device(s) 1850 may be one or more of a variety of input devices such as a mouse, a keyboard, a tracking ball, a voice input device, etc. The output device(s) 1860 may be one or more of a variety of output devices such as a display, a speaker, a printer, etc. The communication unit 1840 allows the computing device 1800 to further communicate with one or more external devices (not shown), such as a storage device and a display device, with one or more devices that allow a user to interact with the computing device 1800, or with any device (such as a network card, a modem, etc.) that allows the computing device 1800 to communicate with one or more other computing devices as needed. Such communication may occur via an input / output (I / O) interface (not shown).

[0240] In some embodiments, instead of being integrated into a single device, some or all of the components of computing device 1800 may be located in a cloud computing architecture. In a cloud computing architecture, the components may be provided remotely and may cooperate to implement the functionality described in this disclosure. In some embodiments, cloud computing provides computing, software, visual data access and storage services without the end user needing to be aware of the physical location or configuration of the systems or hardware providing these services. In various embodiments, cloud computing provides services over a wide area network (such as the Internet) using appropriate protocols. For example, a cloud computing provider provides applications accessible through a web browser or other computing components over a wide area network. The software or components of the cloud computing architecture and the corresponding visual data may be stored on a server in a remote location. The computing resources of the cloud computing environment may be consolidated or distributed across locations in a remote visual data center. The cloud computing infrastructure may act as a single access point for users, but provide services through a shared visual data center. Thus, a cloud computing architecture may be used to provide the components and functionality described herein from a remote service provider. Alternatively, it may be provided from a conventional server or installed directly or in other manner on the client device.

[0241] The computing device 1800 may be used to perform visual data encoding / decoding in embodiments of the present disclosure. The memory 1820 may include one or more visual data encoding modules 1825 having one or more program instructions. These modules are accessible and executable by the processing unit 1810 to perform the functions of the various embodiments described herein.

[0242] In an example embodiment performing visual data encoding, the input device 1850 may receive visual data as input 1870 to be encoded. The visual data may be processed, for example, by the visual data encoding module 1825 to generate an encoded bitstream. The encoded bitstream may be provided as output 1880 via the output device 1860.

[0243] In an example embodiment performing visual data decoding, the input device 1850 may receive an encoded bitstream as input 1870. The encoded bitstream may be processed, for example, by the visual data encoding module 1825 to generate decoded visual data. The decoded visual data may be provided as output 1880 via the output device 1860.

[0244] Although the present disclosure has been particularly shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application as defined by the appended claims. Such variations are intended to be covered by the scope of the present application. Thus, the foregoing description of the embodiments of the present application is not intended to be limiting.

Claims

1. obtaining an intermediate representation of the visual data for conversion between visual data and a bitstream of the visual data, the intermediate representation being different from a quantized latent representation of the visual data; At least one parameter, at least a portion of the quantized latent representation; a prediction of the at least a portion of the quantized latent representation; or a difference between the prediction and the at least a portion of the quantized latent representation; and and performing a composition transformation on the intermediate representation for the transformation; 10. A method for processing visual data, wherein the quantized latent representation is generated based on applying a first neural network to the visual data.

2. Obtaining the intermediate representation comprises: obtaining the at least a portion of the quantized latent representation; updating the at least a portion of the quantized latent representation based on at least one of the at least one parameter, the prediction, or the difference to obtain at least a portion of the intermediate representation; The method of claim 1 , comprising:

3. Updating the at least a portion of the quantized latent representation comprises: scaling the at least a portion of the quantized latent representation with a first parameter of the at least one parameter; updating the at least a portion of the quantized latent representation based on a product of the prediction and a second parameter of the at least one parameter and / or a product of the difference and a third parameter of the at least one parameter; or updating the at least a portion of the quantized latent representation by adding at least a fourth parameter of the at least one parameter; The method of claim 2 , comprising at least one of:

4. The method of claim 2 or 3, wherein the prediction is a mean or the difference is included in a quantized residual latent representation of the visual data.

5. Obtaining the at least a portion of the quantized latent representation comprises:

5. The method of claim 2, further comprising performing an entropy decoding process on the bitstream to obtain the at least a portion of the quantized latent representation.

6. The method of claim 5 , further comprising generating the prediction by using a first model.

7. Obtaining the at least a portion of the quantized latent representation comprises: performing an entropy decoding process on the bitstream to obtain the difference; generating the prediction by using a first model; generating the at least a portion of the quantized latent representation based on the prediction and the difference; The method according to any one of claims 2 to 4, comprising:

8. Generating the prediction comprises:

8. The method of claim 6 or 7, comprising generating predictions of samples in the at least a portion of the quantized latent representation based on at least one reconstructed sample of the quantized latent representation by using the first model.

9. The method according to any one of claims 6 to 8, wherein the first model is a predictive model.

10. The method according to any one of claims 6 to 9, wherein the first model is autoregressive.

11. The method according to any one of claims 6 to 10, wherein the first model comprises a context sub-network or a context model sub-network.

12. 12. The method of claim 2, wherein the at least a portion of the quantized latent representation includes all samples of the quantized latent representation, and the intermediate representation corresponds to a result of the update.

13. updating a further portion of the quantized latent representation based on at least one further parameter different from the at least one parameter; The method of any one of claims 2 to 11, wherein the further portion is distinct from the at least a portion of the quantized latent representation.

14. Obtaining the intermediate representation comprises: The method of claim 1 , comprising generating at least a portion of the intermediate representation based on the prediction and the difference.

15. Generating the at least a portion of the intermediate representation comprises: generating the at least a portion of the intermediate representation by adding a product of the prediction and a first one of the at least one parameter and a product of the difference and a second one of the at least one parameter; or generating the at least a portion of the intermediate representation by adding a product of the prediction and the first parameter, a product of the difference and the second parameter, and a third parameter of the at least one parameter; 15. The method of claim 14, comprising:

16. The method of claim 14 or 15, wherein at least one of the prediction or the difference is generated by using a first model.

17. 17. The method of any one of claims 6 to 8 and 16, wherein the first model is an estimated model.

18. The method of any one of claims 6 to 8 and claims 16 to 17, wherein the first model comprises a neural network based sub-network or an input of the first model comprises the bitstream.

19. 18. The method of claim 6, wherein the first model includes a first sub-network for generating the prediction and a second sub-network for generating statistics.

20. 20. The method of claim 19, wherein the first sub-network is a hyperdecoder sub-network and the second sub-network is a hyperscale decoder sub-network.

21. The method of any one of claims 1 to 20, further comprising determining the at least a portion of the quantized latent representation from the quantized latent representation.

22. Determining the at least a portion of the quantized latent representation from the quantized latent representation comprises: determining whether the at least a portion of the quantized latent representation includes a sample of the quantized latent representation based on at least one of the following: comparing a statistical value corresponding to the sample with a first threshold; a comparison of a second threshold value with a value determined based on the statistical value; or a comparison of the sample index with a third threshold; 22. The method of claim 21.

23. Determining the at least a portion of the quantized latent representation from the quantized latent representation comprises: determining whether the at least a portion of the quantized latent representation includes a sample within a block of the quantized latent representation based on at least one of the following: comparing the statistical value corresponding to each said sample with a fourth threshold; a comparison of a fifth threshold value to a metric determined based on a statistical value corresponding to the sample; or a sixth threshold value and a comparison of the index of each of the samples; 22. The method of claim 21.

24. The method of claim 23 , wherein the metric is an average value, a minimum value, or a maximum value.

25. The sample index is the channel number of said sample, a feature map identifier for the sample; or The spatial coordinates of the sample The method according to any one of claims 22 to 24, wherein the method is one of the following:

26. The first threshold value, The second threshold value, The third threshold value, The fourth threshold value, the fifth threshold value, or The sixth threshold A method according to any one of claims 22 to 25, wherein at least one of, or at least one representation of, is indicated in the bitstream.

27. The method of any one of claims 19 to 26, wherein the statistic is a variance.

28. The method according to any one of claims 19 to 27, wherein the statistic is a variance of a Gaussian probability or the statistic is obtained based on the bitstream by using the second sub-network.

29. A method according to any preceding claim, wherein said at least one parameter or a representation of said at least one parameter is included in said bitstream.

30. The method according to any one of claims 1 to 29, wherein the at least one parameter is a scalar value or a vector different from zero.

31. The method according to any one of claims 1 to 30, wherein the at least one parameter is determined based on a quality metric.

32. The quality metric is: Mean square error, structural similarity (SSIM), or Multi-scale structural similarity (MS-SSIM) 32. The method of claim 31 , comprising at least one of:

33. The method of any one of claims 1 to 32, wherein the at least a portion of the quantized latent representation comprises one or more samples of the quantized latent representation.

34. The synthesis transformation is performed by using a neural network-based sub-network; or the first neural network is used to perform analytical transformations on the visual data; 34. The method according to any one of claims 1 to 33.

35. A method according to any preceding claim, wherein at least one of information regarding whether to apply the method or information regarding how to apply the method is indicated in the bitstream.

36. 35. The method of any one of claims 1 to 34, wherein at least one of the information regarding whether to apply the method or how to apply the method depends on the color format and / or color components of the visual data.

37. The method according to any one of the preceding claims, wherein values ​​included in the bitstream are coded at one of a sequence level, a picture level, a slice level or a block level.

38. A method according to any one of claims 1 to 37, wherein values ​​contained in the bitstream are binarised before being encoded.

39. A method according to any one of claims 1 to 38, wherein values ​​comprised in the bitstream are coded with at least one arithmetic coding context.

40. A method according to any preceding claim, wherein the visual data comprises an image or a video picture.

41. A method according to any preceding claim, wherein the transforming comprises encoding the visual data into the bitstream.

42. A method according to any preceding claim, wherein the converting comprises decoding the visual data from the bitstream.

43. An apparatus for visual data processing comprising a processor and a non-transitory memory having instructions thereon, the instructions, when executed by the processor, causing the processor to perform a method according to any one of claims 1 to 42.

44. A non-transitory computer readable storage medium storing instructions for causing a processor to perform the method of any one of claims 1 to 42.

45. 1. A non-transitory computer readable recording medium storing a bitstream of visual data generated by a method performed by an apparatus for visual data processing, the method comprising: obtaining an intermediate representation of the visual data, the intermediate representation being different from the quantized latent representation of the visual data; At least one parameter, at least a portion of the quantized latent representation; a prediction of the at least a portion of the quantized latent representation; or a difference between the prediction and the at least a portion of the quantized latent representation; and generating the bitstream based on a synthesis transformation on the intermediate representation; The quantized latent representation is generated based on applying a first neural network to the visual data.

46. 1. A method for storing a bitstream of visual data, comprising the steps of: obtaining an intermediate representation of the visual data, the intermediate representation being different from the quantized latent representation of the visual data; At least one parameter, at least a portion of the quantized latent representation; a prediction of the at least a portion of the quantized latent representation; or a difference between the prediction and the at least a portion of the quantized latent representation; and generating the bitstream based on a composition transformation on the intermediate representation; storing the bitstream on a non-transitory computer-readable recording medium; 11. A method for storing a bitstream of visual data, wherein the quantized latent representation is generated based on applying a first neural network to the visual data.

Images