Visual data processing method, device and medium

Abstract

Embodiments of the present disclosure provide a solution for visual data processing. A method for visual data processing is presented. The method comprises: for a conversion between visual data and a bitstream of visual data, obtaining an intermediate representation of the visual data, the intermediate representation being different from a quantized implicit representation of the visual data, a quantized implicit representation and generated based on at least one of: at least one parameter, at least a portion of the quantized implicit representation, a prediction of at least a portion of the quantized implicit representation, or a difference between the prediction and at least a portion of the quantized implicit representation; and performing a synthetic transformation on the intermediate representation for the transformation, where the quantized hidden representation is generated based on applying the first neural network to the visual data.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to visual data processing technology, and more specifically, to neural network-based visual data encoding and decoding. Background Art

[0002] The past decade has witnessed the rapid development of deep learning in various fields, especially in computer vision and image processing. Neural networks were originally invented using interdisciplinary research in neuroscience and mathematics. It has shown strong capabilities in nonlinear transformations and classification. Neural network-based image / video compression technology has made significant progress during the past five years. It is reported that the latest neural network-based image compression algorithms achieve rate-distortion (RD) performance comparable to that of the general video codec (VVC). As the performance of neural images continues to improve, neural network-based video compression has become an actively developing research area. However, it is generally desired to further improve the codec quality of neural network-based image / video codecs. Summary of the invention

[0003] Embodiments of the present disclosure provide a solution for visual data processing.

[0004] In a first aspect, a method for visual data processing is presented. The method comprises: for conversion between visual data and a bitstream of the visual data, obtaining an intermediate representation of the visual data, the intermediate representation being different from a quantized latent representation of the visual data and being generated based on at least one of: at least one parameter, at least a portion of the quantized latent representation, a prediction of at least a portion of the quantized latent representation, or a difference between the prediction and at least a portion of the quantized latent representation; and for conversion, performing a synthesis transform on the intermediate representation, wherein the quantized latent representation is generated based on applying a first neural network to the visual data.

[0005] According to the method of the first aspect of the present disclosure, an intermediate representation different from the quantized implicit representation of the visual data is generated and used for the synthetic transformation. Compared with the conventional solution in which the quantized implicit representation is directly used for the synthetic transformation, the proposed method can at least partially eliminate the artifacts caused by the conventional conversion process, and thus the reconstructed image can be more visually pleasing. In this way, the proposed method can advantageously improve the encoding and decoding quality.

[0006] In a second aspect, a device for visual data processing is provided. The device includes a processor and a non-volatile memory having instructions thereon. The instructions, when executed by the processor, cause the processor to perform the method according to the first aspect of the present disclosure.

[0007] In a third aspect, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores instructions, which enable a processor to execute the method according to the first aspect of the present disclosure.

[0008] In a fourth aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of visual data, the bitstream being generated by a method performed by a visual data processing device. The method includes: obtaining an intermediate representation of the visual data, the intermediate representation being different from a quantized implicit representation of the visual data and being generated based on at least one of the following: at least one parameter, at least a portion of the quantized implicit representation, a prediction of at least a portion of the quantized implicit representation, or a difference between the prediction and at least a portion of the quantized implicit representation; and generating a bitstream based on a synthetic transformation of the intermediate representation, wherein the quantized implicit representation is generated based on applying a first neural network to the visual data.

[0009] In a fifth aspect, a method for storing a bitstream of visual data is provided. The method includes: obtaining an intermediate representation of the visual data, the intermediate representation being different from a quantized implicit representation of the visual data and being generated based on at least one of: at least one parameter, at least a portion of the quantized implicit representation, a prediction of at least a portion of the quantized implicit representation, or a difference between the prediction and at least a portion of the quantized implicit representation; generating a bitstream based on a synthetic transformation of the intermediate representation, and storing the bitstream in a non-transitory computer-readable recording medium, wherein the quantized implicit representation is generated based on applying a first neural network to the visual data.

[0010] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become more apparent through the following detailed description with reference to the accompanying drawings.In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0012] Figure 1 A block diagram illustrating an example data encoding and decoding system according to some embodiments of the present disclosure is shown;

[0013] Figure 2 A typical transform coding scheme is shown;

[0014] Figure 3 An image from the Kodak dataset and different representations of the image are shown;

[0015] Figure 4 The network architecture of the autoencoder implementing the hyper-prior model is shown;

[0016] Figure 5 A block diagram of the combined model is shown;

[0017] Figure 6 The encoding process of the combined model is shown;

[0018] Figure 7 The decoding process of the combined model is shown;

[0019] Figure 8 An alternative encoding process is shown;

[0020] Fig. 9 An alternative decoding process is shown;

[0021] Fig.10 Two possible implementations of the entropy encoding and decoding and quantization process are shown;

[0022] Fig.11 The corresponding decoder implementation for the encoder implementation is shown;

[0023] Fig.12 shows an example decoder implementation according to some embodiments of the present disclosure;

[0024] Fig.13 Another example decoder implementation according to some embodiments of the present disclosure is shown;

[0025] Fig.14 Another example decoder implementation according to some embodiments of the present disclosure is shown;

[0026] Fig.15 shows an example visual data decoding process according to some embodiments of the present disclosure;

[0027] Fig.16 Another example visual data decoding process according to some embodiments of the present disclosure is shown;

[0028] Fig.17 A flowchart of a method for visual data processing according to some embodiments of the present disclosure is shown;

[0029] Fig.18 A block diagram of a computing device is shown in which various embodiments of the present disclosure may be implemented.

[0030] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. DETAILED DESCRIPTION

[0031] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are only for illustrative purposes, and contribute to those skilled in the art to understand and implement the present disclosure, without implying any limitation to the scope of the present disclosure. The disclosure described herein can be realized in various ways different from the modes described below.

[0032] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0033] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example embodiment, it is submitted that it is within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in conjunction with other embodiments, whether or not explicitly described.

[0034] It should be understood that although the terms "first" and "second" etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the example embodiments, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0035] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the terms "include", "including", "have", "have", "include" and / or "include" specify the presence of the features, elements and / or components, etc. when used herein, but do not exclude the presence or addition of one or more other features, elements, components and / or combinations thereof. Example Environment

[0036] Figure 11 is a block diagram illustrating an example visual data encoding and decoding system 100 that may utilize the techniques of the present disclosure. As shown, the visual data encoding and decoding system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a visual data encoding device, and the destination device 120 may also be referred to as a visual data decoding device. In operation, the source device 110 may be configured to generate encoded visual data, and the destination device 120 may be configured to decode the encoded visual data generated by the source device 110. The source device 110 may include a visual data source 112, a visual data encoder 114, and an input / output (I / O) interface 116.

[0037] Visual data source 112 may include a source such as a visual data capture device. Examples of a visual data capture device include, but are not limited to, an interface for receiving visual data from a visual data provider, a computer graphics system for generating visual data, and / or combinations thereof.

[0038] The visual data may include one or more images or one or more pictures of a video. The visual data encoder 114 encodes the visual data from the visual data source 112 to generate a bitstream. The bitstream may include a bit sequence that forms a coded representation of the visual data. The bitstream may include a coded picture and associated visual data. The coded picture is a coded representation of a picture. The associated visual data may include a sequence parameter set, a picture parameter set, and other grammatical structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The coded visual data may be directly transmitted to the destination device 120 via the network 130A via the I / O interface 116. The coded visual data may also be stored on a storage medium / server 130B for access by the destination device 120.

[0039] Destination device 120 may include an I / O interface 126, a visual data decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain the encoded visual data from source device 110 or storage medium / server 130B. The visual data decoder 124 may decode the encoded visual data. The display device 122 may display the decoded visual data to the user. The display device 122 may be integrated with the destination device 120, or may be external to the destination device 120, which is configured to be connected to an external display device interface.

[0040] The visual data encoder 114 and the visual data decoder 124 may operate according to a visual data codec standard, such as a video codec standard or a still image codec standard, as well as other current and / or further standards.

[0041] Some exemplary embodiments of the present disclosure will be described in detail below. It should be understood that section titles are used in this document to promote ease of understanding, and the embodiments disclosed in a section are not limited to that section. In addition, although certain embodiments are described with reference to general video codecs or other specific visual data codecs, the disclosed technology is also applicable to other codec technologies. In addition, although some embodiments describe the coding and decoding steps in detail, it should be understood that the corresponding step decoding of the undo coding and decoding will be implemented by the decoder. In addition, the term visual data processing includes visual data coding or compression, visual data decoding or decompression, and visual data transcoding, in which visual data is represented from one compression format to another compression format or with different compression bit rates. 1. Brief Overview A neural network based image and video compression method includes an autoregressive subnetwork, wherein the latent representation of the image is first processed by the autoregressive network, then modified according to an addition or multiplication term, and finally processed by a synthesis network to obtain a reconstructed image. 2. Introduction The past decade has witnessed the rapid development of deep learning in various fields, especially in computer vision and image processing. From the great success of deep learning techniques in the field of computer vision, many researchers have shifted their attention from traditional image / video compression techniques to neural image / video compression techniques. Neural networks were originally invented using interdisciplinary research in neuroscience and mathematics. It shows strong capabilities in nonlinear transformation and classification. Image / video compression techniques based on neural networks have made significant progress during the past five years. It is reported that the latest neural network-based image compression algorithms achieve RD performance comparable to Versatile Video Codec (VVC), the latest video codec standard developed by the Joint Video Experts Group (JVET) with experts from MPEG and VCEG. With the continuous improvement of the performance of neural images, neural network-based video compression has become an actively developed research area. However, neural network-based video codec is still in its infancy because the problem itself is difficult. 2.1. Image / Video Compression Image / video compression generally refers to computing techniques that compress images / videos into binary codes for easy storage and transmission. The binary codes may or may not support lossless reconstruction of the original image / video, known as lossless compression and lossy compression. Most research is devoted to lossy compression, since lossless reconstruction is not required in most scenarios. Usually, the performance of image / video compression algorithms is evaluated from two aspects, namely compression ratio and reconstruction quality. The compression ratio is directly related to the number of binary codes, the worse it is; the reconstruction quality is measured by comparing the reconstructed image / video with the original image / video, the better it is. Image / video compression technology can be divided into two branches, classical video codec methods and neural network-based video compression methods. Classical video coding schemes use transform-based solutions, in which researchers exploit the statistical dependencies of these latent variables (e.g., DCT or wavelet coefficients) in the quantization domain through carefully hand-written entropy codes. There are two forms of neural network-based video compression: neural network-based codec tools and end-to-end neural network-based video compression. The former is embedded into existing classical video codecs as codec tools and is only used as part of the framework, while the latter is a separate framework developed based on neural networks without relying on classical video codecs. Over the past three decades, a series of classic video codec standards have been developed to accommodate the increased visual content. The international standards organization ISO / IEC has two expert groups, the Joint Photographic Experts Group (JPEG) and the Moving Picture Experts Group (MPEG), and ITU-T also has its own Video Codec Experts Group (VCEG), which is used for the standardization of image / video codec technologies. Influential video codec standards released by these organizations include JPEG, JPEG 2000, H.262, H.264 / AVC, and H.265 / HEVC. After H.265 / HEVC, the Joint Video Experts Group (JVET) formed by MPEG and VCEG has been working on a new video codec standard, the Versatile Video Codec (VVC). The first version of VVC was released in July 2020. It is reported that VVC achieves an average bit rate reduction of 50% over HEVC at the same visual quality. Neural network-based image / video compression is not a new invention, as there are many researchers working on image codecs based on neural networks. However, the network architecture is relatively shallow and the performance is not satisfactory. Benefiting from the support of large amounts of data and powerful computing resources, neural network-based methods are better utilized in various applications. Currently, neural network-based image / video compression has shown promising improvements, proving its feasibility. However, the technology is still far from mature and many challenges need to be addressed. Neural Networks A neural network, also known as an artificial neural network (ANN), is a computational model used in machine learning techniques that is typically composed of multiple processing layers, and each layer is composed of multiple simple but nonlinear basic computational units. One benefit of such deep networks is believed to be the ability to process data with multiple layers of abstraction and transform the data into different types of representations. Note that these representations are not manually designed; instead, the deep network including the processing layers is learned from massive amounts of data through general machine learning programs. Deep learning eliminates the need for manual representations and is therefore believed to be useful, particularly for processing originally unstructured data, such as acoustic and visual signals, which has been a long-standing problem in the field of artificial intelligence. 2.3. Neural Networks for Image Compression Existing neural networks for image methods can be divided into two categories, namely pixel probability modeling and autoencoders. The former belongs to the predictive encoding and decoding strategy, while the latter is a transformation-based solution. Sometimes, these two methods are combined in the literature. 2.3.1. Pixel Probabilistic Modeling According to Shannon's information theory, the best method for lossless coding can achieve the minimum coding rate -log2p(x), where p(x) is the symbol probability. Many lossless coding methods have been developed in the literature, among which arithmetic coding is considered to be one of the best methods. Given a probability distribution p(x), arithmetic coding ensures that the coding rate is as close to its theoretical limit -log2p(x) as possible without considering rounding errors. Therefore, the remaining question is how to determine the probability, which is very challenging for natural images / videos due to the reduction in dimensionality. Following the predictive coding strategy, one way to model p(x) is to predict pixel probabilities one by one in raster scan order based on previous observations, where x is an image. p(x)=p(x1)p(x2|x1)…p(x i |x1,…,x i-1 )…p(x m×n |x1,…,x m×n-1 ) (1) Where m and n are the height and width of the image, respectively. The previous observation is also called the context of the current pixel. When the image is large, it may be difficult to estimate the conditional probability, so a simplified approach is to limit the scope of its context. p(x)=p(x1)p(x2|x1)…p(x i |x i-k ,…,x i-1 )…p(x m×n |x m×n-k ,…,x m×n-1 ) (2)where k is a predefined constant that controls the scope of the context. It should be noted that the conditions may also take into account sample values ​​of other color components. For example, when encoding and decoding RGB color components, the R sample depends on previously encoded and decoded pixels (including R / G / B samples), the current G sample may be encoded and decoded based on the previously encoded and decoded pixels and the current R sample, and for encoding and decoding the current B sample, the previously encoded and decoded pixels and the current R and G samples may also be considered. Neural networks were initially introduced for computer vision tasks and have been shown to be effective in regression and classification problems. Therefore, it has been proposed to use neural networks to estimate a given its context x1,x2,…,x i-1 The probability p(x i ). For a binary image, that is, x i ∈{-1,+1}, pixel probability is proposed. The Neural Autoregressive Distribution Estimator (NADF) is designed for pixel probability modeling and is a feedforward network with a single hidden layer. Similar work has been presented in existing designs, where the feedforward network also has connections that skip the hidden layer and parameters are also shared p(x i |x1,…,x i-1 ). Experiments are performed on the binarized MNIST dataset. In the existing design, NADF is extended to the real-valued model RNADE, where the probabilities are derived using a mixture of Gaussians. Their feed-forward network also has a single hidden layer, but the hidden layer is rescaled to avoid saturation and uses rectified linear units (ReLU) instead of sigmoids. In the existing design, NADF and RNADE are improved by rearranging the order of pixels and utilizing deeper neural networks. Designing advanced neural networks plays an important role in improving pixel probability modeling. In existing designs, multidimensional long short-term memory (LSTM) is proposed, which is used for probability modeling together with conditional Gaussian scaled mixture. LSTM is a special kind of recurrent neural network (RNN) and is proven to be good at modeling sequential data. Spatial variants of LSTM are used for images in existing designs. Several different neural networks are studied, including RNN and CNN, namely PixelRNN and PixelCNN. In PixelRNN, two variants of LSTM are proposed, called row LSTM and diagonal BiLSTM, of which the latter is specifically designed for images. PixelRNN contains residual connections to help train deep neural networks with up to 12 layers. In PixelCNN, masked convolution is used to adapt to the shape of the context. Compared with previous work, PixelRNN and PixelCNN are more specialized for natural images: they treat pixels as discrete values ​​(e.g., 0, 1, ..., 255) and predict a multinomial distribution of discrete values; they process color images in RGB color space; they work well on the large-scale image dataset ImageNet. In the existing design, Gated PixelCNN is proposed to improve PixelCNN and achieve comparable performance to PixelRNN, but with much lower complexity. In the existing design, PixelCNN++ has the following improvements on PixelCNN: discretized logistic mixture likelihood is used instead of multinomial distribution of 256 channels; downsampling is used to capture structure at multiple resolutions; additional shortcut connections are introduced to speed up training; dropout is used for regularization; RGB is combined for one pixel. In the existing design, PixelSNAIL is proposed, in which causal convolution is combined with self-attention. Most of the above methods directly model the probability distribution in the pixel domain. Some researchers also try to model the probability distribution as a conditional distribution based on explicit or implicit representations. That is, it can be estimated Where h is an additional condition and p(x)=p(h)p(x|h), which means that the modeling is divided into an unconditional part and a conditional part. The additional condition can be image label information or a high-level representation. 2.3.2. Automatic Codec Autoencoders are derived from the well-known work proposed by Hinton and Salakhutdinov. The method is trained for dimensionality reduction and consists of two parts: encoding and decoding. The encoding part converts the high-dimensional input signal into a low-dimensional representation, usually with a reduced spatial size but with a larger number of channels. The decoding part tries to recover the high-dimensional input from the low-dimensional representation. Autoencoders enable automatic learning of representations and eliminate the need for hand-crafted features, which is also considered one of the most important advantages of neural networks. Figure 2 A typical transform coding scheme is shown. The original image x is analyzed by the network g a Transform to achieve the implicit representation y. The implicit representation y is quantized and compressed into bits. The number of bits R is used to measure the encoding and decoding rate. Then, the synthetic network g s The quantized implicit representation Perform inverse transform to obtain the reconstructed image By using the function g p Transform x and x to compute the distortion in perceptual space. It is intuitive to apply autoencoder networks to lossy compression. This only requires encoding the learned latent representations from a trained neural network. However, adapting the autoencoder to images is not simple because the original autoencoder is not optimized for compression, so directly using the trained autoencoder is not efficient. Additionally, there are other major challenges: First, the low-dimensional representation should be quantized before encoding, but quantization is not differentiable, which is required in backpropagation when training neural networks. Second, the goals in the compression scenario are different because both distortion and rate need to be considered. Estimation of the bitrate is challenging. Third, practical image coding and decoding schemes need to support variable bitrate, scalability, encoding / decoding speed, and interoperability. To address these challenges, many researchers have actively contributed to this field. Prototype autoencoder for image compression in Figure 2 It can be regarded as a transform coding strategy. a (x) Transform the original image x, where y is the latent representation that will be quantized and decoded. The synthesis network will inverse transform the quantized latent representation to obtain the reconstructed image The training framework is based on the rate-distortion loss function, i.e., where D is x and The distortion between R is the quantized representation The bit rate is calculated or estimated, and λ is the Lagrange multiplier. It should be noted that D can be calculated in the pixel domain or the perceptual domain. All existing research works follow this prototype, and the difference may only be the network structure or loss function. In terms of network structure, RNN and CNN are the most widely used architectures. In the RNN-related category, Toderici et al. proposed a general framework for variable bitrate images using RNN. They used binary quantization to generate codes and did not consider the bitrate during training. The framework actually provides scalable encoding and decoding capabilities, and RNNs with convolutional and deconvolutional layers are reported to perform well. Then, Toderici et al. proposed an improved version by upgrading the encoder using a neural network similar to PixelRNN to compress the binary code. It is reported that the method performs better than JPEG on the Kodak image dataset using the MS-SSIM evaluation metric. Johnston et al. further improved the RNN-based solution by introducing hidden state preprocessing. In addition, an SSIM weighted loss function is designed and a spatially adaptive bitrate mechanism is enabled. Their results on the Kodak image dataset are better than BPG using MS-SSIM as the evaluation metric. Covell et al. support spatially adaptive bitrate by training a fault-tolerant stop encoding and decoding RNN. Ballé et al. proposed a general framework for rate-distortion optimized images. Integer codes are generated using multivariate quantization, and the code rate is considered during training, i.e., the loss is a joint rate-distortion cost, which can be either mean squared error (MSE) or other forms. They add random uniform noise during training to motivate quantization, and use the differential entropy of the noise code as a proxy for the code rate. They use generalized dinormalization (GDN) as the network structure, which includes a linear mapping followed by a nonlinear parameter normalization. The effectiveness of GDN for images has been verified in existing designs. Ballé et al. then proposed an improved version, in which they used 3 convolutional layers, each followed by a downsampling layer and a GDN layer as a forward transform. Therefore, they used 3 layers of inverse GDN, each followed by an upsampling layer and a convolutional layer to motivate the inverse transform. Additionally, an arithmetic coding method is designed to compress integer codes. It is reported that its performance is better than JPEG and JPEG 2000 on the Kodak dataset. In addition, Ballé et al. improved the method by designing a scale hyper-prior in the automatic codec. They used a subnet h a Transform the latent representation y into z = h a (y), and z is quantized and sent as side information. Accordingly, the inverse transform is performed through the subnet h s Implementation, try to extract the quantized side information Decode the quantized The standard deviation of further used in the arithmetic coding process of . On the Kodak image set, their method is slightly worse than BPG in terms of PSNR. D.Minnen et al. further exploited the structure of the residual space by introducing an autoregressive model to estimate the standard deviation and mean. In the existing design, Z.Cheng et al. used a Gaussian mixture model to further remove redundancy in the residual. It is reported that on the Kodak image set, when PSNR is used as the evaluation metric, its performance is comparable to VVC. 2.3.3. Hyper-prior model In the transform codec approach to image compression, the encoder subnetwork (Section 2.3.2) uses a parametric analysis transform The image vector x is transformed into a latent representation y, which is then quantized as In the form of is a discrete value that can be losslessly compressed and transmitted as a bit sequence using entropy coding techniques, such as arithmetic coding. As from Figure 3 The middle left image and the middle right image can clearly see that the elements There is a significant spatial dependence between them. It is worth noting that their scales (middle right image) appear to be coupled in the spatial domain. In the existing design, another set of random variables is introduced to capture spatial dependencies and further reduce redundancy. In this case, the image compression network Figure 4 Shown in. exist Figure 4 The left hand side of the model is the encoder g a and decoder g s (Explained in Section 2.3.2). The right hand side is used to obtain In this architecture, the encoder and decoder take the input image x and feed it into g a , thus generating a response y with a spatially varying standard deviation. The response y is sent to h a , to summarize the standard deviation distribution of z. Then z is quantized as is compressed and transmitted as side information. The encoder then uses the quantized vector To estimate the spatial distribution of the standard deviation σ, and use it to compress and transmit the quantized image representation The decoder first recovers the compressed signal Then it uses h s to obtain σ, which provides the correct probability estimate for successful recovery Then Feed g s to obtain a reconstructed image. When the super encoder and super decoder are added to the image compression network, the quantized latent The airspace redundancy is reduced. Figure 3 The rightmost image in corresponds to the quantized latent when using a super encoder / decoder. Compared to the middle right image, the spatial redundancy is significantly reduced because the samples of the quantized latent are less correlated. Figure 3 An image from the Kodak dataset and different representations of the image are shown. Figure 3 The leftmost image in the figure shows an image from the Kodak dataset. Figure 3 The middle left image in shows a visualization of the latent representation y of the image. Figure 3 The middle right image in shows the standard deviation σ of the latent quantity. Figure 3 The rightmost image in shows the latent y after introducing the super (super encoder and decoder) network. Figure 4 The network architecture of the autoencoder implementing the super-prior model is shown. The image autoencoder network is shown on the left, and the right corresponds to the super-prior subnetwork. The analysis and synthesis transformations are denoted as g a and g s Q represents quantization, AE and AD represent arithmetic encoder and arithmetic decoder respectively. The super prior model consists of two sub-networks, the super encoder (with h a denoted) and super decoder (with h s The super prior model generates quantized super implicit It includes the implicit quantity Information about the probability distribution of samples. is included in the bitstream and Sent together to the receiving end (decoder). 2.3.4. Context Model Although the hyper-prior model improves the quantized implicit , but additional improvement can be obtained by exploiting autoregressive models that predict quantized latent quantities from their causal context (context model). The term autoregressive means that the output of a process is later used as its input. For example, the context model subnetwork generates one sample of the latent quantity, which is later used as input to get the next sample. Existing designs utilize a joint architecture where both the super-prior model subnetwork (super-encoder and super-decoder) and the context model subnetwork are utilized. The super-prior and context models are combined to learn a quantized latent The probability model of is then used for entropy coding and decoding. Figure 5As shown, the output of the context sub-network and the super decoder sub-network is called the sub-network combination of entropy parameters, which generates the mean μ and scale ρ (or variance) parameters of the Gaussian probability model. The Gaussian probability model is then used to encode the quantized implicit samples into the bitstream with the help of the arithmetic encoder-decoder (AE) module. In the decoder, the Gaussian probability model is used to obtain the quantized implicit samples from the bitstream through the arithmetic decoder (AD) module. Figure 5 The block diagram of the combined model is shown. The combined model jointly optimizes the autoregressive component, which estimates the probability distribution of the delay from its causal context (context model) together with the hyperprior and the underlying autoencoder. The real-valued latent representation is quantized (Q) to create a quantized latent and quantified super-implicit It is compressed into a bitstream using an arithmetic encoder (AE) and decompressed by an arithmetic decoder (AD). The highlighted areas correspond to the components executed by the receiving end (ie, the decoder) to recover the image from the compressed bitstream. Usually, latent samples are modeled as Gaussian distribution or Gaussian mixture model (not limited to). Figure 5 , the context model and the hyper-prior are used jointly to estimate the probability distribution of latent samples. Since a Gaussian distribution can be defined by a mean and a variance (also called σ or scale), the joint model is used to estimate the mean and variance (denoted as μ and ρ). 2.3.5. Encoding and decoding process using joint autoregressive hyper-prior model Figure 5 Corresponding to the existing technical compression method. In this section and the next section, the encoding and decoding processes will be described respectively. Figure 6 The encoding process is depicted. The input image is first processed using the encoder subnetwork. The encoder transforms the input image into a transformed representation called the latent, denoted by y. Then y is input to the quantizer block denoted by Q to obtain the quantized latent after is converted into a bitstream (bits1) using an arithmetic coding block (denoted as AE). The arithmetic coding block converts the Each sample of is converted into a bit stream (bits1). The module super encoder, context, super decoder and entropy parameter sub-network are used to estimate the probability distribution of quantized latent samples. The hidden quantity y is input to the super encoder, and the super encoder outputs the super hidden quantity (denoted by z). The super hidden quantity is then quantized as The arithmetic encoding and decoding (AE) module is used to generate a second bit stream (bits2). The decomposition entropy module generates a probability distribution, which is used to encode the quantized super-hidden quantity into the bit stream. The quantized super-hidden quantity includes information about the quantized hidden quantity information about the probability distribution of . The entropy parameter subnetwork generates the quantized latent The information generated by the entropy parameters usually includes the mean μ and scale (or variance) σ parameters that are used together to obtain a Gaussian probability distribution. The Gaussian distribution of a random variable x is defined as Where parameter μ is the mean or expected value of the distribution (also its median and mode), and parameter ρ is its standard deviation (or variance, or scale). To define a Gaussian distribution, the mean and variance need to be determined. In existing designs, the entropy parameter module is used to estimate the mean and variance values. The subnetwork superdecoder generates part of the information used by the entropy parameter subnetwork, and the other part of the information is generated by an autoregressive module called the context module. The context module generates information about the probability distribution of samples of the quantized latent quantity using samples that have been encoded by the arithmetic coding (AE) module. This is usually a matrix consisting of many samples. or The index to indicate the sample depends on the matrix Dimensions of . Sample Encoded by the AE one by one, usually using raster scan order. In raster scan order, the rows of the matrix are processed from top to bottom, where samples in a row are processed from left to right. In this case (where the AE encodes the samples into the bitstream using raster scan order), the context module generates the same samples as the samples using samples encoded previously in raster scan order. The information generated by the context module and the super decoder is combined by the entropy parameter module to generate the latent quantity Probability distribution encoded into the bitstream (bits1). Finally, as a result of the encoding process, the first bitstream and the second bitstream are sent to a decoder. Note that other names can be used for the above modules. In the above description, Figure 6 All the elements in are collectively called encoders. The analytical transformation that transforms the input image into a latent representation is also called an encoder (or autoencoder). 2.3.6. Decoding process using the joint autoregressive hyper-prior model Figure 7The decoding process is depicted separately. In the decoding process, the decoder first receives the first bit stream (bits1) and the second bit stream (bits2) generated by the corresponding encoder. Bits2 is first decoded by the arithmetic decoding (AD) module by using the probability distribution generated by the decomposition entropy subnetwork. The decomposition entropy module usually generates the probability distribution using a predetermined template, such as a predetermined mean and variance value in the case of a Gaussian distribution. The output of the arithmetic decoding process of bits2 is the quantized super-implicit The AD process is the inverse of the AE process applied in the encoder. The AE and AD processes are lossless, which means that the quantized super-hidden data generated by the encoder is Can be reconstructed at the decoder without any changes. In obtaining After that, it is processed by the super decoder, whose output is fed to the entropy parameter module. The three subnetworks, context, super decoder, and entropy parameters used in the decoder are the same as those in the encoder. Therefore, the exact same probability distribution can be obtained in the decoder as in the encoder, which is essential for reconstructing the quantized latent information without lossless As a result, the quantized implicit quantity obtained in the encoder can be obtained in the decoder The same version of . After the entropy parameter subnetwork obtains the probability distribution (e.g., mean and variance parameters), the arithmetic decoding module decodes the quantized latent samples one by one from the bitstream bits1. From a practical point of view, the autoregressive model (context model) is serial in nature and therefore cannot be accelerated using techniques such as parallelization. Finally, the fully reconstructed quantized implicit Input to the composite transform (in Figure 7 The decoder module is used to obtain the reconstructed image. In the above description, Figure 7 All elements in are collectively referred to as decoders. The synthetic transform that converts the quantized latent quantity into the reconstructed image is also called a decoder (or automatic decoder). 2.3.7. Alternative Encoding Procedure Using Joint Autoregressive Hyperprior Model Alternative implementations of the encoder can be found in Figure 8 Depicted in Figure 6 The difference between the encoders depicted in is how the quantized implicit quantity is obtained and what is encoded into the bitstream. exist Figure 6 In the quantified implicit The samples of are obtained by quantizing the latent quantity y. In addition, The samples of are included in the bitstream by an entropy encoder (or arithmetic encoder). It should be noted that the arithmetic encoder is a special case of an entropy encoder. exist Figure 8 In the quantified implicit According to the mean value μ and the quantized residual implicit The average value is obtained by using The previously acquired samples of are recursively acquired. The quantized residual implicit The result is obtained by subtracting μ from the implicit y and then quantizing it. Finally, the arithmetic encoder module (AE) encodes the quantized residual implicit into the bitstream. 2.3.8. Alternative decoding process using joint autoregressive hyper-prior model An alternative implementation of the decoder can be found in Fig. 9 Depicted in Figure 7 The decoders depicted in differ in how the quantized implicits are obtained and what is obtained from the bitstream. exist Fig. 9 In the example, the quantized residual implicit It is obtained from the bitstream by an arithmetic decoder (AD or entropy decoder). Using the mean μ and the quantized residual The average value is obtained by the prediction module using previously obtained samples of y. 2.4. Neural Networks for Video Compression Similar to traditional video codec technology, neural image compression is used as the basis for intra-frame compression in neural network-based video compression, so the development of neural network-based video compression technology is later than that of neural network-based image compression, but more efforts are needed to solve the challenges caused by its complexity. Since 2017, some researchers have been working on neural network-based video compression schemes. Compared with image compression, video compression requires effective methods to remove redundancy between pictures. Inter-picture prediction is then a key step in these works. Motion estimation and compensation are widely adopted, but have not been implemented by trained neural networks until recently. Research on neural network-based video compression can be divided into two categories according to the target scenario: random access and low latency. In the case of random access, decoding is required to start from any point in the sequence, usually dividing the entire sequence into multiple independent segments, and each segment can be decoded independently. In the case of low latency, it aims to reduce the decoding time, so that subsequent frames are usually decoded using only the previous frame in time as a reference frame. 2.4.1. Low latency Chen et al. first proposed a video compression scheme with a trained neural network. They first divided the video sequence frames into blocks, and each block will choose one of the two available modes, intra-frame codec or inter-frame codec. If intra-frame codec is selected, there is an associated autoencoder for compressing the block. If inter-frame codec is selected, motion estimation and compensation are performed using traditional methods, and the trained neural network will be used for residual compression. The output of the autoencoder is directly quantized and encoded by the Huffman method. Chen et al. proposed another neural network-based video codec scheme with PixelMotionCNN. Frames are compressed in temporal order and each frame is divided into blocks that are compressed in raster scan order. Each frame will first be extrapolated with the previous two reconstructed frames. When a block is to be compressed, the extrapolated frame is sent to PixelMotionCNN along with the context of the current block to derive the latent representation. The residual is then compressed by a variable bitrate image scheme. The performance of this scheme is comparable to H.264. Lu et al. proposed a real-time end-to-end neural network-based video compression framework in which all modules are implemented with neural networks. The scheme accepts the current frame and the previous reconstructed frame as input, and the optical flow will be derived as motion information using a pre-trained neural network. The motion information will be warped with the reference frame, followed by a neural network that generates motion compensated frames. Two separate neural autoencoders are used to compress the residual and motion information. The entire framework is trained with a single rate-distortion loss function. It achieves better performance than H.264. Rippel et al. proposed a video compression scheme based on advanced neural networks. It inherits and extends the traditional video codec scheme of neural networks with the following main features: 1) only one autoencoder is used to compress motion information and residuals; 2) multiple frames and multiple optical flows are used for motion compensation; 3) online states are learned and propagated through subsequent frames over time. This scheme achieves better performance than the HEVC reference software in MS-SSIM. J.Lin et al. proposed a video compression framework based on an extended end-to-end neural network. In this scheme, multiple frames are used as references. Therefore, it is possible to provide a more accurate prediction of the current frame by using multiple reference frames and associated motion information. Additionally, motion field prediction is used to remove motion redundancy in the temporal channel. A post-processing network is also introduced in this work to remove reconstruction artifacts from the previous process. The scheme outperforms H.265 in terms of both PSNR and MS-SSIM. Eirikur et al. proposed a scale-spatial flow to replace the commonly used optical flow by adding a scale parameter. It is reported that better performance than H.264 is achieved. Z.Hu et al. proposed a multi-resolution representation for optical flow. Specifically, the motion estimation network generates multiple optical flows with different resolutions and lets the network learn which one to choose under the loss function. The performance is slightly improved and better than H.265. 2.4.2. Random Access Wu et al. proposed a neural network-based video compression scheme with frame interpolation. Key frames are first compressed with a neural image compressor, and the remaining frames are compressed in a hierarchical order. They perform motion compensation in the perceptual domain, i.e., feature maps are derived at multiple spatial scales of the original frame, and the motion is used to warp the feature maps to be used for the image. The method is reported to have performance comparable to H.264. Djelouah et al. proposed a method for interpolation-based video compression, where the interpolation model combines motion information compression and image synthesis, and the same autoencoder is used for both the image and the residual. Amirhossein et al. proposed a neural network-based video compression method based on variational autoencoders and deterministic encoders. Specifically, the model consists of an autoencoder and an autoregressive prior. Unlike previous methods, this method accepts group of pictures (GOP) as input and encodes latent representations by considering temporal correlations, thereby incorporating a 3D autoregressive prior. The method provides performance comparable to H.265. 2.5. Preliminary knowledge Almost all natural images / videos are in digital format. A grayscale digital image can be represented as in is a set of pixel values, m is the image height, and n is the image width. For example, a common setting is in this case, Therefore, pixels can be represented by 8-bit integers. Uncompressed grayscale digital images have 8 bits per pixel (bpp), and the number of bits after compression is definitely less. Color images are usually represented in multiple channels to record color information. For example, in the RGB color space, an image can be represented as Three independent channels store red, green, and blue information respectively. Similar to an 8-bit grayscale image, an uncompressed 8-bit RGB image has 24bpp. Digital images / videos can be represented in different color spaces. Neural network-based video compression schemes are mainly developed in the RGB color space, while traditional codecs usually use the YUV color space to represent video sequences. In the YUV color space, the image is decomposed into three channels, namely Y, Cb, and Cr, where Y is the brightness component and Cb / Cr are the chrominance components. Since the human visual system is less sensitive to the chrominance components, Cb and Cr are usually downsampled for pre-compression. A color video sequence consists of multiple color images, called frames, to record scenes at different time stamps. For example, in the RGB color space, a color video can be represented as X = {x0, x1, …, x t ,…,x T-1}, where T is the number of frames in the video sequence, If m=1080, n=1920, And the video has 50 frames per second (fps), then the data rate of the uncompressed video is 1920×1080×8×3×50=2,488,320,000, about 2.32 Gbps, which requires a lot of storage and thus must be compressed before transmission over the Internet. Typically, for natural images, lossless methods can achieve a compression ratio of about 1.5 to 3, which is clearly lower than required. Therefore, lossy compression is developed to achieve further compression ratios, but at the expense of distortion. Distortion can be measured by calculating the average squared difference between the original image and the reconstructed image, i.e., the mean square error (MSE). For grayscale images, MSE can be calculated using the following equation. Therefore, the quality of the reconstructed image compared to the original image can be measured by the Peak Signal-to-Noise Ratio (PSNR): in yes The maximum possible value in , for 8-bit grayscale images, it is equal to 255. There are other quality assessment metrics, such as structural similarity (SSIM) and multi-scale structural similarity (MS-SSIM). To compare different lossless compression schemes, the compression ratio is compared at a given resulting rate or vice versa. However, to compare different lossy compression methods, both the rate and the reconstruction quality must be considered. For example, calculating the relative rate at multiple quality levels and then taking the average rate is a common approach; the average relative rate is called Bjontegaard's delta rate (BD-rate). There are other important aspects for evaluating image / video codec schemes, including encoding / decoding complexity, scalability, robustness, etc. 2.5.1. Quantization and Entropy Coding and Decoding The terms quantization and entropy coding can be defined as follows: In mathematics and digital signal processing, quantization is the process of mapping input values ​​from a large (usually continuous) set to output values ​​in a (countably) smaller set, usually with a finite number of elements. Rounding and truncation are classic examples of quantization processes. Almost all digital signal processing involves quantization to some extent, since the process of representing a signal in digital form usually involves rounding. Quantization also forms the core of essentially all lossy compression algorithms. The quantization equation can be as follows: in, is the quantized value, y is the sample to be quantized, and Δ is the quantization step size. For example, according to the above equation 1, if the quantization step size increases, the quantized value will become smaller and fewer bits will be required to encode the quantized samples. In other words, an increased quantization step size results in a coarser quantization. On the other hand, a smaller Δ (usually a positive number) results in a finer quantization and a higher number of bits to encode the quantized samples. Encoded into the bitstream. Entropy coding is a general lossless data compression method that encodes symbols using a number of bits that is inversely proportional to the symbol probability. This term is used to refer to some compression algorithms that are based on the probability distribution of the source symbols. Huffman coding and arithmetic coding are two typical representatives of this type of algorithm. In order to perform entropy coding, the statistical properties of the symbols (such as the probability distribution) must be understood. Fig.10 Two possible implementations of entropy coding, decoding and quantization in NN-based image compression are shown. Described from the encoder perspective. exist Fig.10 In the figure, two possible implementations of an encoder for an NN-based image compression method are described. The encoder for the NN-based image compression method consists of four modules. The first module is an analysis transform. The analysis transform transforms the input image into an implicit representation y. The transformed representation y is usually easier to compress than the input image because the analysis transform is designed to eliminate redundancy in the input image signal. The second module is an estimation module, which estimates the statistical properties of the implicit representation. The statistical properties may include a mean (expressed as μ), a variance (expressed as σ), or higher-order moments. The third module is a quantization module, which converts a continuous implicit representation signal into a quantized discrete value. Finally, the fourth module is an entropy codec (expressed as EC), which converts the quantized symbol values ​​into a bit stream using the statistical properties generated by the estimation module. For the encoder, there are two slightly different implementation alternatives, depending on how the mean value μ is used. In the first implementation, the entropy coding module receives the mean value and other statistics (such as variance σ) as input and uses it to encode the quantized hidden symbols into the bitstream. In the second implementation alternative, the mean value is subtracted from the hidden symbols y to obtain residual samples w. The residual samples are then quantized by the quantization module to obtain quantized residual samples The entropy coding module converts the quantized residual samples into a bitstream using the statistical information generated by the estimation module. The statistical information may include, for example, the variance σ. Fig.11 Shown for Fig.10 The corresponding decoder implementation for the encoder implementation shown in . Fig.11 Two possible alternative implementations of a decoder in a NN-based video compression method are described. According to the first alternative, the estimation module estimates the mean μ and other statistical properties (e.g., variance σ). The variance and optionally other statistical information are used by an entropy codec (denoted as EC) to decode the bitstream into quantized residual samples. Then the mean μ and the quantized residual samples are estimated are added together to obtain the quantized latent samples Finally, it is transformed into a reconstructed picture through a synthetic transformation. The first alternative decoder is implemented in Fig.11 is depicted in the upper flow chart. According to a second alternative, the estimation module estimates the mean μ and other statistical properties (e.g. variance σ). The mean, variance and optionally other statistical information are used by the entropy codec (denoted EC) to decode the bitstream into quantized hidden samples Finally, it is transformed into a reconstructed picture through a synthetic transform. The second alternative decoder is implemented in Fig.11 is depicted in the bottom flow chart. In the above, the quantized latent sample is given by It should be noted that a sample is not necessarily a scalar value, it may be a vector and may contain multiple elements. In the rest of the application, the sample can be represented by or In the latter, “:” is used to indicate the presence of a third dimension and to emphasize that a sample has multiple elements. 3. Question 3.1. Core Issues In the prior art, autoregressive networks are used to predict the quantized latent The term autoregression indicates self-reference or self-dependence. The samples of are used to generate predictions, which are then used to obtain The next sample of A sample of depends on at least one previously acquired The process is autoregressive. once All samples of are obtained, and they are processed by the synthesis transform to obtain the reconstructed image. The problem in the prior art is that the same The samples of are used by the prediction module and the synthesis transform. This can be Figure 7 and Fig. 9 was observed in Used by the "decoder" module (in other words, the synthesis transform) and in the loop including the "context" module. The "context" module is used to predict the next Part of the prediction module for samples. Therefore, in the prior art, the same latent sample is used to obtain a reconstructed image via a synthetic transform and prediction of the next latent sample. However, sample values ​​that lead to good predictions do not always lead to the most visually pleasing reconstructed images. For example, humans may subjectively prefer reconstructed images with sharp details and high contrast. On the other hand, latent samples that lead to sharper reconstructed images require the transmission of more side information (increasing bit rate) because they are usually difficult to predict. 4. Detailed solution The detailed solutions below should be considered as examples to explain the general concept. These solutions should not be interpreted in a narrow way. In addition, these solutions can be combined in any way. 4.1. Objectives of the project The goal of image compression is to maximize the quality of the output image while reducing the number of bits to be sent (bitrate). This solution proposes a method for improving the quality of reconstructed pictures without increasing the bitrate of neural network based image and video compression methods. 4.2. Solution Core According to this solution, the quantized implicit quantity used by the synthetic transform The samples are different from those used by the prediction module. The core of the scheme includes the following steps: 1. Obtain quantified hidden quantity The first sample, 2. Use the first sample prediction The second sample of (i.e., getting a predicted value), 3. Use the predicted value to obtain The second sample, 4. Modify the first sample to obtain the modified quantized latent 5. Processing by synthetic transformation to obtain a reconstructed image. In other words, the quantized implicit quantity is modified after it is used for prediction and before the synthetic transform is applied. 4.2.1. Example implementation of the solution According to this scheme, the reconstructed image is acquired according to the following steps: First, the following steps are used to autoregressively obtain samples of the quantized latent quantity: 1. Obtain a sample of mean μ by using the prediction module. The input of the prediction module is the previously obtained quantized latent of the sample. 2. Optionally, get the residual from the bitstream of the sample. 3. Use mean and / or residual samples to obtain quantized latent quantities New samples. 4. Use mean and / or residual samples and quantized latent quantities Get the modified quantized implicit New samples. In obtaining the modified quantized implicit After , the synthesis transform is applied to obtain the reconstructed picture. 4.2.1.1. Second alternative According to this scheme, the reconstructed image is acquired according to the following steps: First, the quantized latent samples are obtained autoregressively using the following steps: 1. Obtain a sample of the mean value μ by using the prediction module. The input of the prediction module is the previously obtained quantized latent of the sample. 2. Optionally, get the residual from the bitstream of the sample. 3. Use mean and / or residual samples to obtain quantized latent quantities New samples. In getting After the sample, use and the mean μ (or residual ) obtains the modified quantized implicit The synthetic transform is applied to the modified quantized latent to obtain the reconstructed image. 4.3. Differences between the solution and existing technologies In existing NN-based image and video coding systems, the reconstructed image is obtained according to the following steps: First, the following steps are used to autoregressively obtain samples of the quantized latent quantity: 1. Obtain a sample of the mean value μ by using the prediction module. The input of the prediction module is the previously obtained quantized latent of the sample. 2. Optionally, get the residual from the bitstream of the sample. 3. Use mean and / or residual samples to obtain quantized latent quantities New samples. Once the quantified hidden All samples of are obtained and the synthesis transform is applied to obtain the reconstructed picture. In the prior art, the same quantized implicit samples are used by the prediction module and the synthesis transform. On the other hand, according to this solution, the modified quantized implicit samples used by the synthesis transform are different from the quantized implicit samples used in the prediction module. 4.4. Solution details Three possible implementations of the solution are described in detail in this section. 4.4.1. Alternative Implementation 1 Fig.12 A possible implementation of a NN-based image compression scheme is shown. The decoder perspective is depicted. Fig.12 One of the possible implementations of the solution is depicted. According to the solution, first a bit stream (denoted by bits1 or bits) is received. The entropy decoder (ED) module decodes the bit stream to obtain the quantized implicit The first sample is represented as Then it is used by the prediction module to obtain a second sample of the statistical parameters μ2 and / or σ2. The statistical parameters can be (but are not limited to) a mean parameter μ or a variance parameter σ. The statistical parameters can also include parameters that can be used to describe the probability distribution. After the second statistical parameter sample is obtained, it is used by the prediction module to obtain a second sample of the quantized latent quantity This process is repeated until all samples of the quantized latent quantity are acquired. As explained in the above paragraph, the first sample of the quantized latent quantity is used by the prediction module, the output of which is used to obtain the second sample of the quantized latent quantity. Due to the recursive nature of the process, the quantized latent quantity The acquisition of is an autoregressive process. According to this solution, the prediction of the second sample is performed using an autoregressive neural network or a sub-network (prediction module). The prediction process can predict the mean μ or the variance parameter σ. According to the first alternative implementation of this scheme, the quantized implicit The samples of are obtained through the entropy decoding process. The average value μ is obtained by the prediction module using the previously obtained The entropy codec module then uses the average value to obtain The next sample of . The sample of the quantized residual can be obtained according to the following equation. exist After the sample of is obtained, it is modified to obtain the quantized latent sample The quantized latent samples are then used by a synthesis transform to obtain a reconstructed picture. 4.4.2. Alternative Implementation 2 Fig.13 Another possible implementation of a NN-based image compression scheme is shown. The decoder perspective is depicted. In addition to how to obtain the quantized latent and the quantized residual latent, the second alternative implementation of the solution ( Fig.13 According to the second alternative, the output of the entropy decoding module (ED) is the quantized residual implicit It is used to obtain according to and the average value μ is obtained, where the average value μ is obtained by The samples of are obtained by applying the prediction module. Since the average value is obtained using the previously obtained samples, the prediction module is an autoregressive process. In addition, the prediction module includes a network or sub-network based on a neural network. exist After the sample of is obtained, it is modified to obtain the quantized latent sample The quantized latent samples are then used by the synthesis transform to obtain the reconstructed picture. 4.4.3. Alternative Implementation 3 Fig.14 Another possible implementation of the solution in NN-based image compression is shown. The decoder perspective is depicted. Another possible implementation is in Fig.14 According to this alternative implementation, the quantized hidden samples are multiplied by a multiplier after being used by the prediction module to obtain modified quantized hidden samples. The modified quantized hidden samples are used by a synthetic transform (denoted as "decoder") to obtain a reconstructed image. In all the above alternative implementations, the value of the multiplier (scale1 and / or scale2) or an indication thereof may be obtained from the bitstream. It should be noted that the term decoder may be used to refer to both the "synthetic transform" and the entire decoding process (including the synthetic transform, the prediction module and the entropy decoding module). 4.4.4. Further details of the solution: According to the scheme, the modified quantized implicit It can be obtained according to the following equation. m1, m2, and m3 are multipliers, which can be vectors of scalar numbers. or or or or or or or or … Furthermore, alternatively, the quantized implicit It can be obtained according to the following equation: a1 is the addition term, which can be a scalar number or a numeric vector. The value of the addition term a1 may be included in a bitstream by an encoder and decoded from the bitstream by a decoder. The addition term a1 and the multiplication terms m1, m2, and m3 can be applied together. An example is as follows: or or or or or or or or … According to an example implementation, The modified samples are based on (but not limited to) the quantized latent The quantized implicit residual At least two of the and the average value (μ) are obtained. For example, the modified quantized implicit Can be based on and is obtained. Or it can be used and μ are obtained. Or it can be based on and μ are obtained. It should be noted that according to this scheme, the modified quantized implicit At least one sample of The entropy decoder module can use the variance and mean parameters to obtain a probability distribution, such as a Gaussian distribution. For example, a Gaussian distribution can be obtained using the following equation: Where the variance σ and mean parameter μ are used in the equation. The probability distribution is then used to obtain the quantized residual latent or quantized latent. Once the modified quantized implicit is used to obtain a reconstructed image by applying a synthetic transform. The synthetic transform can be a neural network based subnetwork consisting of convolutional or deconvolutional layers. This is the core of the solution, i.e., using quantized latent To obtain a sample Additional samples and further modifications before synthetic transformation In other words, the core of the program includes the steps of: 1. Obtain quantified hidden quantity The first sample, 2. Use the first sample prediction The second sample, 3. Use the prediction to obtain The second sample, 4. Modify the first sample to obtain a modified quantized latent 5. Processing by synthetic transformation Get the reconstructed image. The value of the multiplier (or an indication thereof) may be included in the bitstream and decoded by the decoder. Quantified implicit quantity Can be selectively modified based on rules. For example, a rule can include one of the following: If the value of the associated probability parameter is less than (or greater than) the threshold, the quantized implicit The samples of are modified. For example, if the value of the variance parameter σ[c,i,j] is less than the threshold, then the samples in the corresponding position (for example, ) can be modified to obtain If σ[c,i,j] is greater than or equal to the threshold, then This can be done by setting it equal to is acquired. -The function of the probability parameter can be used to determine Which sample is modified. For example, the equation F(σ[c,i,j]) < thr can be used to determine whether the corresponding sample at the position [c,i,j] is modified. o The probability parameter can be a variance value. o The function can be in the form of a probability distribution or a cumulative probability distribution. An example is a Gaussian probability distribution (e.g., ) or a Laplace distribution or the like. - Quantized latent quantity The sample is modified according to its index value. For example, if the value of the index c is equal to the predetermined index value K, then the sample can be modified. Or in another example, if the index i is greater than the predetermined index value T, then the sample can be modified. The index c can indicate the "channel number" of the "feature map identifier". The indices i and j indicate the spatial coordinates of the sample. The values K and / or T can be included in the bitstream and retrieved by the decoder from the bitstream. Or the values of K and / or T can be predetermined. Combinations of checking whether the index is "greater than", "less than", "equal to" the value can be used to determine whether the sample belongs to the first subset. 4.4.5. Encoding and decoding process according to the scheme In the encoder, is obtained in a similar manner as in the decoder. For example, is obtained according to one of the following equations. Or Or Or Or Or Or Or Or Or Or Or Or Or Or Or Or or … The encoder determines the value of the multiplier based on a quality metric. In one example, it can test different values ​​of the multiplier to determine which result has the highest quality. The quality metric can be mean square error or MS-SSIM, etc. It can then include the multiplier (or an indication thereof) in the bitstream so that the decoder can replicate the result obtained by the encoder. 4.5. Advantages of the solution By modifying the quantized latent quantity before performing a synthetic transform, artifacts caused by the conversion process may be at least partially eliminated, and thus the reconstructed image may be more visually pleasing. 5. Examples 1. Decoder embodiment: A method for decoding an image or video, comprising at least one of the following steps: -Use the bitstream and prediction module to obtain the quantized implicit The first sample - Use the first sample to obtain the quantized latent The second sample, -According to the quantified implicit The first sample of , obtains the modified quantized implicit The first sample, Using modified quantized implicit Get the reconstructed image, the modified quantized latent Can be used with composite transforms. 2. According to Example 1, Get the modified quantized implicit The first sample includes: - Get the addition item, -According to the quantified implicit The first sample of and the addition term, obtain the modified quantized implicit The first sample. 3. According to embodiment 2; The step of obtaining the addition term comprises: obtaining the quantized residual from the bit stream The first sample. 4. According to embodiment 3; Quantized residual The first sample is used to obtain the quantized latent The first sample. 5. According to embodiment 2; The step of obtaining the added term comprises obtaining a first sample of the mean value μ using a prediction module. 6. According to embodiment 5; The first sample of the mean value μ is used to obtain the quantized latent quantity The first sample. 7. According to embodiments 5 and 6; The first sample of the mean μ is obtained by the prediction module using the quantized implicit At least one other sample is acquired. 8. According to embodiment 2; The step of obtaining the addition term comprises obtaining a first sample of the mean value μ from the bit stream. 9. According to embodiments 5 to 8; The first sample of the mean value μ is used to obtain the quantized latent quantity The first sample. 10. According to embodiment 2; The value of the addition term (or an indication thereof) is obtained from the bitstream. 11. According to embodiment 2; The additive term is the quantized implicit quantity And the modified quantized implicit according to or is obtained, where m1 is not equal to 1 and m2 is not equal to 0. 12. According to embodiment 1; The first sample of the modified quantized implicit quantity is calculated according to is acquired. 13. According to any of the above embodiments, The steps of obtaining the addition term include: Subtract the mean μ of the samples from the samples of . 14. According to any of the above embodiments, The steps of obtaining the addition term include: The first sample of is subtracted from the mean value μ. 15. According to any of the above embodiments, Quantified implicit quantity and the modified quantized implicit comprising at least a second sample, wherein the quantized latent The first samples are used by the prediction module to obtain a second sample of the mean value μ. 16. According to embodiment 15, The second sample of the mean μ is used to obtain the quantized latent The second sample. 17. According to embodiment 16, Modified quantized implicit The second sample is obtained by using the quantized implicit A second sample of is acquired. 18. According to any of the above embodiments, The prediction module includes a neural sub-network. 19. According to any of the above embodiments, The prediction module is an autoregressive subnetwork. 20. According to any of the above embodiments, The prediction module receives the previously acquired quantized latent As input, process the input. 21. According to any of the above embodiments, The prediction module includes a context sub-network or a super-decoder sub-network. 22. According to any of the above embodiments, The multiplier (or an indication thereof) is included in the bitstream. 23. According to any of the above embodiments, The multiplier is a scalar value different from 0. 24. According to any of the above embodiments, The multiplier is a vector. 25. According to any of the above embodiments, The composite transform is a neural sub-network. 26. According to any of the above embodiments, a value of at least one segment of control information is included, which can determine whether and / or how to apply any of the above methods. 27. Whether and / or how to apply any of the above methods may depend on conditions, such as color format and / or color components. 28. According to any of the above embodiments, any value included in the bitstream may be encoded or decoded at a sequence level / picture level / slice level / block level. 29. According to any of the above embodiments, any value included in the bitstream may be binarized before being encoded or decoded. 30. According to any of the above embodiments, any value included in the bitstream can be encoded using at least one arithmetic codec context.

[0042] More details of embodiments of the present disclosure related to neural network-based visual data encoding and decoding are described below.As used herein, the term "visual data" may refer to an image, a picture in a video, or any other visual data suitable for being encoded and decoded.

[0043] As described above, in existing designs, the quantized latent representation of visual data is directly used in a synthetic transform to reconstruct the visual data. However, conventional conversion processes may be affected by artifacts, which make the reconstructed visual data less visually pleasing.

[0044] In order to solve the above problems and some unmentioned problems, a visual data processing solution as described below is disclosed.

[0045] Fig.15 An example visual data decoding process 1500 according to some embodiments of the present disclosure is shown. For example, the visual data decoding process 1500 may be performed by Figure 1 The visual data decoder 124 shown executes. It should be understood that the visual data decoding process 1500 may also include additional blocks not shown, and / or may omit the blocks shown. The scope of the present disclosure is not limited in this regard.

[0046] like Fig.15 As shown, the bitstream can be input to an entropy decoder 1520. At the entropy decoder 1520, an entropy decoding process can be performed on the bitstream to obtain a quantized residual implicit representation of the visual data (in Fig.15 In the expression ). It should be noted that the quantized residual implicit representation may also be referred to as quantization of the residual implicit representation. Similarly, the quantized implicit representation may also be referred to as quantization of the implicit representation. The residual implicit representation and / or the quantized residual implicit representation may indicate a difference between an implicit representation of the visual data and a prediction of the implicit representation.

[0047] As an example and not limitation, the entropy decoding process can be based on the decomposition entropy sub-network ( Fig.15 The probability distribution information generated by the decomposition entropy sub-network (not shown) is executed. In some embodiments, the decomposition entropy sub-network can generate probability distribution information by using a predetermined template, for example, by using a predetermined mean and variance value in the case of a Gaussian distribution. The entropy decoding process performed by the entropy decoder 1520 can be an arithmetic decoding process, a Huffman decoding process, etc.

[0048] At block 1518, the quantized implicit representation of the visual data ( Fig.15 In the expression ) can be obtained by predicting the quantized residual implicit representation and the implicit representation generated by the prediction model 1522 ( Fig.15The prediction model 1522 may be implemented using a neural network. It should be noted that in the present application, a model may also be referred to as a module. That is, a prediction model may also be referred to as a prediction module, and an estimation model may also be referred to as an estimation module. In some embodiments, the process may be determined at a sample level. In other words, a sample of a quantized implicit representation may be obtained by summing a sample corresponding to a quantized residual implicit representation and a prediction of the sample.

[0049] In some embodiments, prediction model 1522 may be autoregressive. In one example, prediction model 1522 may include a context subnetwork and a prediction fusion subnetwork ( Fig.15 quantized implicit representation At least one reconstructed sample of can be input to the context subnetwork. At least one reconstructed sample is previously obtained and used to generate a quantized latent representation The prediction of the current sample is not adjusted. The context subnetwork may generate intermediate information based on at least one reconstructed sample. As an example and not limitation, the intermediate information may reflect an average value of at least one reconstructed sample. The prediction fusion subnetwork may generate a prediction of the current sample based on the intermediate information. In one example, the prediction may indicate an average value of the prediction of the current sample. The obtained prediction may be compared with the quantized residual implicit representation. The above process can be repeated to reconstruct the quantized implicit representation All samples of .

[0050] It should be understood that the context sub-network 1426 may also be referred to as a context model, a context model sub-network, etc. In addition, the prediction fusion sub-network may also be referred to as a fusion sub-network, a prediction sub-network, etc. In addition, the prediction model 1522 may also be implemented in any other suitable manner, for example, a multi-level context model may be adopted, and the prediction fusion sub-network may be removed. The scope of the present disclosure is not limited in this regard.

[0051] At block 1514, the quantized implicit representation can be updated to obtain an intermediate representation of the visual data (in Fig.15 In the expression ). The intermediate representation is different from the quantized implicit representation. In some embodiments, the updating process can be performed at the sample level. In other words, the updating process is performed by sample. Alternatively, the quantized implicit representation After all samples of are obtained, the updating process can be performed on the entire quantized latent representation.

[0052] In some embodiments, at least a portion of the quantized implicit representation may be updated based on at least one of: at least one parameter, a prediction of at least a portion of the quantized implicit representation, or a difference between the prediction and the quantized implicit representation. By way of example and not limitation, at least one parameter may be determined based on a quality metric at the encoder side or the decoder side. The quality metric may include mean square error, structural similarity (SSIM), multi-scale structural similarity (MS-SSIM), and the like. In one example, at least one parameter may be a scalar value or vector different from zero. At least one parameter may be signaled in the bitstream. Alternatively, an indication of at least one parameter may be signaled in the bitstream.

[0053] In some embodiments, at least a portion of the quantized implicit representation may be updated by scaling the at least a portion with a first parameter. Additionally or alternatively, at least a portion of the quantized implicit representation may be updated based on a product of the prediction and a second parameter and / or a product of the difference and a third parameter. In some additional or alternative embodiments, at least a portion of the quantized implicit representation may be updated by adding a fourth parameter. This is described in detail at Section 4.4.4 above, where possible formulas for the update process are listed for illustrative and non-limiting purposes.

[0054] In some embodiments, different samples of the quantized implicit representation may be adjusted in different ways (such as using different parameters). Additionally or alternatively, a set of samples of the quantized implicit representation may not be updated. In other words, samples of the quantized implicit representation may be selectively updated.

[0055] In some embodiments, a grouping process may be performed at update block 1514 for selective updating in order to determine at least a portion of the quantized implicit representation that needs to be updated or updated in a particular manner. In one example, the grouping process may be performed at the sample level. In other words, the determination of the at least a portion is performed for each sample of the quantized implicit representation. Alternatively, the grouping process may be performed at the block level. More specifically, the samples of the quantized residual implicit representation may be divided into a plurality of blocks. Each of the plurality of blocks may have a predetermined size of N times M, such as 8×8. Each of N and M is a positive integer. N and M may be signaled in the bitstream. Samples may be grouped based on a plurality of blocks. In other words, the determination of the at least a portion is performed for each of the plurality of blocks, and the samples in the same block are divided into the same set.

[0056] In some embodiments, for a sample of a quantized implicit representation, if the statistical value corresponding to the sample is less than a first threshold value, it can be determined that the at least one portion includes the sample, and the sample can be grouped into a first set. Otherwise, it can be determined that the sample will not be included in the at least one portion, and the sample can be grouped into a second set different from the first set. As an example and not limitation, the statistical value can be a variance, such as the variance of a Gaussian probability. The variance can be generated based on a bitstream using a superscale decoder subnetwork. It should be understood that any other suitable statistical value can also be used, such as a mean value or a standard deviation. In addition, the statistical value can also be referred to as a statistical parameter, a probability parameter, a probability distribution parameter, etc. The scope of the present disclosure is not limited in this regard.

[0057] Alternatively, for a sample of the quantized implicit representation, if the statistical value corresponding to the sample is greater than another threshold value that may be the same as or different from the first threshold value, it may be determined that the at least one portion includes the sample, and the sample may be grouped into the first set. Otherwise, it may be determined that the sample will not be included in the at least one portion, and the sample may be grouped into the second set. The above comparison may be performed for each sample of the quantized residual implicit representation to obtain the at least one portion.

[0058] It should be understood that the at least a portion can be determined in any other suitable manner, such as based on a comparison between a threshold and a value determined based on a statistical value, or based on a comparison between a threshold and an index of the sample. For example, the index of the sample can indicate the channel number of the sample, the feature map identifier of the sample, or the spatial coordinates of the sample. This is described in detail at the above section 4.4.4.

[0059] In some alternative embodiments, the grouping process may be performed at the block level. In this case, for samples in the quantized implicit representation block, if the statistical value corresponding to each sample in the block is less than a fourth threshold, it may be determined that the at least one portion includes all samples in the block, and these samples may be grouped into a first set. Otherwise, it may be determined that these samples will not be included in the at least one portion, and these samples may be grouped into a second set different from the first set.

[0060] Alternatively, for samples in a block of the quantized implicit representation, if the statistical value corresponding to each sample in the block is greater than another threshold value that may be the same as or different from the fourth threshold value, it may be determined that the at least one portion includes all samples in the block, and these samples may be grouped into the first set. Otherwise, it may be determined that these samples will not be included in the at least one portion, and these samples may be grouped into the second set. The above comparison may be performed for each block of the quantized residual implicit representation to obtain the at least one portion.

[0061] It should be understood that the at least a portion can be determined in any other suitable manner, such as based on a comparison between a threshold value and a metric determined based on a statistical value corresponding to the sample, or based on a comparison between a threshold value and an index of each sample. As an example, the metric can be an average value, a minimum value, a maximum value, a specific function, etc. In an example, at least one of the threshold values ​​can be indicated in the bitstream. Alternatively, at least one indication of at least one of the threshold values ​​can be indicated in the bitstream.

[0062] Alternatively, it may be determined sample by sample or block by block whether one or more samples in a block need to be updated, and after the determination, the corresponding samples in the block are updated. As an example and not a limitation, for a sample of a quantized implicit representation, if the statistic corresponding to the sample is less than a threshold value, it may be determined that the sample will be updated. Then, the sample may be updated based on the above-described update process. Similarly, for a sample in a quantized implicit representation block, if the statistic corresponding to each sample in the block is less than a threshold value, it may be determined that the sample in the block will be updated. Then, these samples may be updated based on the above-described update process. It should be noted that this determination process may also be performed based on any other suitable parameters and / or values, such as a value determined based on a statistic or an index of a sample. The scope of the present disclosure is not limited in this respect.

[0063] After at least a portion of the quantized implicit representation is obtained, the above-mentioned update process may be performed on it. It should be understood that at least a portion of the quantized implicit representation may include one or more samples, or even all samples of the quantized implicit representation. In the case where at least a portion of the quantized implicit representation includes all samples of the quantized implicit representation, and the intermediate representation corresponds to the result of updating the at least a portion.

[0064] At synthesis transform block 1512, the intermediate representation A synthesis transformation is performed to obtain reconstructed visual data 1510. For example, the synthesis transformation can be performed by a sub-network based on a neural network. It should be noted that at least a portion of the samples used for the synthesis transformation are different from the samples used for the prediction model 1522.

[0065] In view of the foregoing, an intermediate representation different from the quantized latent representation of the visual data is generated and used for the synthetic transform. Compared with the conventional solution in which the quantized latent representation is directly used for the synthetic transform, the proposed method can at least partially eliminate the artifacts caused by the conventional conversion process, and thus the reconstructed image can be more visually pleasing. In this way, the proposed method can advantageously improve the encoding and decoding quality.

[0066] Fig.16Another example visual data decoding process 1600 according to some embodiments of the present disclosure is shown. For example, the visual data decoding process 1600 may be performed by Figure 1 The visual data decoder 124 shown executes. It should be understood that the visual data decoding process 1600 may also include additional blocks not shown, and / or may omit the blocks shown. The scope of the present disclosure is not limited in this regard.

[0067] like Fig.16 As shown, the bitstream can be input to an entropy decoder 1620. At the entropy decoder 1620, an entropy decoding process can be performed on the bitstream to obtain a quantized residual implicit representation of the visual data (in Fig.16 In the expression ). The entropy decoding process may be performed based on the first statistical value generated by the estimation model 1622. The estimation model 1622 may also be implemented using a neural network. As an example and not limitation, the first statistical value may be the variance (in Fig.16 denoted as σ in ), such as the variance of Gaussian probability. It should be understood that any other suitable statistical value may also be used, such as the mean or standard deviation. The scope of the present disclosure is not limited in this regard.

[0068] In addition to the first statistical value, the estimation model 1622 also generates a second statistical value (in Fig.16 In some embodiments, the second statistic may indicate a prediction of an implicit representation of the visual data. By way of example and not limitation, the second statistic may be an average value, such as an average value of Gaussian probabilities. In one example, the first statistic and the second statistic may be jointly generated by the estimation model 1622. Alternatively, the estimation model 1622 may include a first subnetwork for generating the first statistic and a second subnetwork for generating the second statistic. By way of example and not limitation, the first subnetwork may be a superscale decoder subnetwork and the second subnetwork may be a superdecoder subnetwork.

[0069] At generation block 1614, an intermediate representation of the visual data (in Fig.16 In the expression ) can be implicitly represented based on the second statistical value μ and the quantized residual is generated. It should be noted that since the second statistical value μ indicates the prediction of the implicit representation, it may also be referred to as “prediction” below. Similarly, since the quantized residual implicit representation Indicates the difference between the prediction and the implicit representation, so it may also be referred to as the "difference" below. In some embodiments, at least a portion of the intermediate representation may be generated by adding the product of the prediction and the first parameter and the product of the difference and the second parameter. Alternatively, at least a portion of the intermediate representation may be generated by adding the product of the prediction and the first parameter, the product of the difference and the second parameter, and the third parameter. This is described in detail at Section 4.4.4 above, where the implicit representation for the prediction μ and the quantized residual is listed for illustrative and non-limiting purposes. Generating Intermediate Representation possible equations.

[0070] It should be noted that since the prediction μ and the quantized residual implicitly represent The sum is the quantized implicit representation Therefore, the above generation process can be regarded as Fig.15 The update process described in relation to update block 1514 in FIG. It can also be generated in a similar manner to the selective update described above.

[0071] At synthesis transform block 1612, the intermediate representation A synthesis transformation is performed to obtain reconstructed visual data 1610. For example, the synthesis transformation can be performed by a neural network based sub-network.

[0072] In view of the foregoing, an intermediate representation different from the quantized latent representation of the visual data is generated and used for the synthetic transform. Compared with the conventional solution in which the quantized latent representation is directly used for the synthetic transform, the proposed method can at least partially eliminate the artifacts caused by the conventional conversion process, and thus the reconstructed image can be more visually pleasing. In this way, the proposed method can advantageously improve the encoding and decoding quality.

[0073] Despite the above Fig.15 and Fig.16 Two exemplary visual data decoding processes are described, but it should be understood that any other suitable variations of the visual data decoding process are also conceivable in view of the present disclosure, such as Fig.12 and Fig.14 Two additional exemplary visual data decoding processes are illustratively shown in FIG.

[0074] In addition, although the proposed method is described from the perspective of a decoder, it may also be implemented at a codec, for example, to reconstruct the encoded visual data. The scope of the present disclosure is not limited in this regard. It should be understood that the above illustrations and / or examples are described for descriptive purposes only. The scope of the present disclosure is not limited in this regard. The embodiments of the present disclosure should be considered as examples for explaining general concepts and should not be interpreted in a narrow manner. In addition, these embodiments may be applied individually or in combination in any manner.

[0075] Fig.17 1 shows a flow chart of a method 1700 for visual data processing according to some embodiments of the present disclosure. The method 1700 may be implemented during conversion between visual data and a bit stream of visual data. Fig.17 As shown, method 1700 begins at 1702, where an intermediate representation of visual data is obtained. The intermediate representation is different from a quantized implicit representation of the visual data. In addition, the intermediate representation is generated based on at least one of the following: at least one parameter, at least a portion of the quantized implicit representation, a prediction of at least a portion of the quantized implicit representation, or a difference between the prediction and at least a portion of the quantized implicit representation. The quantized implicit representation is generated based on applying a first neural network to the visual data.

[0076] In some embodiments, the visual data may be analyzed and transformed by a first neural network to obtain an implicit representation of the visual data. The visual data may include one or more pictures in an image or video. The first statistic may be subtracted from the implicit representation to obtain a residual implicit representation. The first statistic may be generated by a second neural network and indicate a prediction of the implicit representation. As an example and not limitation, the first statistic may be an average of a probability distribution, such as a Gaussian probability distribution. For example, a probability distribution may describe a probability distribution of values ​​of one or more samples of the implicit representation.

[0077] At 1704, a synthetic transform is performed on the intermediate representation for conversion. In one example, the conversion may include encoding the visual data into a bitstream. Alternatively or additionally, the conversion may include decoding the visual data from the bitstream. It should be understood that the above description is described for illustrative purposes only. The scope of the present disclosure is not limited in this regard.

[0078] In view of the foregoing, an intermediate representation different from the quantized latent representation of the visual data is generated and used for the synthetic transform. Compared with the conventional solution in which the quantized latent representation is directly used for the synthetic transform, the proposed method can at least partially eliminate the artifacts caused by the conventional conversion process, and thus the reconstructed image can be more visually pleasing. In this way, the proposed method can advantageously improve the encoding and decoding quality.

[0079] In some embodiments, at least a portion of a quantized implicit representation may be obtained at 1702. Furthermore, at least a portion of the quantized implicit representation may be generated based on at least one of at least one parameter, a prediction, or a difference to obtain at least a portion of an intermediate representation.

[0080] In some embodiments, at least a portion of the quantized implicit representation may be updated by scaling at least a portion of the quantized implicit representation by a first parameter of the at least one parameter. Additionally or alternatively, at least a portion of the quantized implicit representation may be updated based on a prediction of the at least one parameter and a product of a second parameter and / or a difference of the at least one parameter and a third parameter. Additionally or alternatively, at least a portion of the quantized implicit representation may be updated by adding at least a fourth parameter of the at least one parameter.

[0081] In some embodiments, the prediction may be an average value. The difference may be included in a quantized residual implicit representation of the visual data. In some embodiments, an entropy decoding process may be performed on the bitstream to obtain at least a portion of the quantized implicit representation. In some embodiments, the method may further include generating the prediction by using a first model. In some embodiments, the difference may be obtained by performing an entropy decoding process on the bitstream. The prediction may be generated by using the first model. At least a portion of the quantized implicit representation may be generated based on the prediction and the difference.

[0082] In some embodiments, a prediction of a sample in at least a portion of the quantized latent representation can be generated based on at least one reconstructed sample of the quantized latent representation using a first model. In one example, the first model can be a prediction model. As an example and not limitation, the first model can be autoregressive. For example, the first model can include a context subnetwork or a context model subnetwork.

[0083] In some embodiments, at least a portion of the quantized implicit representation may include all samples of the quantized implicit representation, and the intermediate representation corresponds to the updated result. In some further embodiments, the method may further include: updating another portion of the quantized implicit representation based on at least one further parameter different from the at least one parameter. The another portion is different from the at least a portion of the quantized implicit representation.

[0084] In some embodiments, at 1702 , at least a portion of an intermediate representation may be generated based on the prediction and the difference.

[0085] In some embodiments, at least a portion of the intermediate representation may be generated by summing the product of the prediction and a first parameter of the at least one parameter and the difference of the at least one parameter and the product of the second parameter. Alternatively, at least a portion of the intermediate representation may be generated by summing the product of the prediction and the first parameter, the difference and the product of the second parameter and the third parameter of the at least one parameter.

[0086] In some embodiments, at least one of the prediction or the difference can be generated by using a first model. In one example, the first model can be an estimation model. For example, the first model can include a subnetwork based on a neural network. The input of the first model can include a bitstream. In some embodiments, the first model can include a first subnetwork for generating a prediction and a second subnetwork for generating a statistical value. By way of example and not limitation, the first subnetwork can be a super decoder subnetwork, and the second subnetwork can be a superscale decoder subnetwork.

[0087] In some embodiments, the method may further include determining at least a portion of the quantized implicit representation from the quantized implicit representation. In one example, whether the at least a portion of the quantized implicit representation includes a sample of the quantized implicit representation may be determined based on at least one of the following: a comparison between a first threshold and a statistical value corresponding to the sample, a comparison between a second threshold and a value determined based on the statistical value, or a comparison between a third threshold and an index of the sample.

[0088] Alternatively, whether at least a portion of the quantized implicit representation includes a sample in a block of the quantized implicit representation may be determined based on at least one of the following: a comparison between a fourth threshold and a statistic corresponding to each sample, a comparison between a fifth threshold and a metric determined based on the statistic corresponding to the sample, or a comparison between a sixth threshold and an index of each sample. In some embodiments, the metric may be an average, a minimum, or a maximum. The index of the sample may indicate one of the following: a channel number of the sample, a feature map identifier of the sample, a spatial coordinate of the sample.

[0089] In some embodiments, at least one of the following thresholds or an indication of at least one of the following may be indicated in the bitstream: a first threshold, a second threshold, a third threshold, a fourth threshold, a fifth threshold, or a sixth threshold. In some embodiments, the statistical value may be a variance. For example, the statistical value may be a variance of a Gaussian probability. The statistical value may be obtained based on the bitstream by the second subnetwork.

[0090] In some embodiments, the at least one parameter or an indication of the at least one parameter may be included in the bitstream. For example, the at least one parameter may be a scalar value or vector different from zero. In one example, the at least one parameter may be determined based on a quality metric. By way of example and not limitation, the quality metric may include at least one of: mean square error, structural similarity (SSIM), or multi-scale structural similarity (MS-SSIM).

[0091] In some embodiments, at least a portion of the quantized latent representation may include one or more samples of the quantized latent representation.In some embodiments, the synthesis transformation may be performed by using a neural network based sub-network.

[0092] In some embodiments, at least one of the following may be indicated in the bitstream: information about whether to apply the method, or information about how to apply the method. In some alternative embodiments, at least one of the following may depend on the color format and / or color components of the visual data: information about whether to apply the method, or information about how to apply the method.

[0093] In some embodiments, the values ​​included in the bitstream may be encoded at one of the following: sequence level, picture level, slice level, or block level. In some embodiments, the values ​​included in the bitstream may be binarized before the bitstream may be encoded. In some embodiments, the values ​​included in the bitstream may be encoded using at least one arithmetic codec context.

[0094] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of visual data, and the bitstream is generated by a method performed by a visual data processing device. The method includes: obtaining an intermediate representation of the visual data, which is different from a quantized implicit representation of the visual data and is generated based on at least one of the following: at least one parameter, at least a portion of the quantized implicit representation, a prediction of at least a portion of the quantized implicit representation, or a difference between the prediction and at least a portion of the quantized implicit representation; and generating a bitstream based on a synthetic transformation of the intermediate representation, wherein the quantized implicit representation is generated based on applying a first neural network to the visual data.

[0095] According to some other embodiments of the present disclosure, a method for storing a bitstream of visual data is provided. The method includes: obtaining an intermediate representation of the visual data, the intermediate representation is different from a quantized implicit representation of the visual data and is generated based on at least one of the following: at least one parameter, at least a portion of the quantized implicit representation, a prediction of at least a portion of the quantized implicit representation, or a difference between the prediction and at least a portion of the quantized implicit representation; generating a bitstream based on a synthetic transformation of the intermediate representation, and storing the bitstream in a non-transitory computer-readable recording medium, wherein the quantized implicit representation is generated based on applying a first neural network to the visual data.

[0096] Embodiments of the present disclosure may be described according to the following clauses, features of which may be combined in any reasonable way.

[0097] Clause 1. A method for visual data processing, comprising: for a conversion between visual data and a bitstream of the visual data, obtaining an intermediate representation of the visual data, the intermediate representation being different from a quantized implicit representation of the visual data and being generated based on at least one of: at least one parameter, at least a portion of the quantized implicit representation, a prediction of the at least a portion of the quantized implicit representation, or a difference between the prediction and the at least a portion of the quantized implicit representation; and for the conversion, performing a synthetic transform on the intermediate representation, wherein the quantized implicit representation is generated based on applying a first neural network to the visual data.

[0098] Clause 2. A method according to clause 1, wherein obtaining the intermediate representation comprises: obtaining at least a portion of the quantized implicit representation; and updating at least a portion of the quantized implicit representation based on at least one of the at least one parameter, the prediction, or the difference to obtain at least a portion of the intermediate representation.

[0099] Clause 3. A method according to clause 2, wherein updating the at least part of the quantized implicit representation comprises at least one of the following: scaling the at least part of the quantized implicit representation using a first parameter of the at least one parameter; updating the at least part of the quantized implicit representation based on a product of the prediction and a second parameter of the at least one parameter and / or a product of the difference and a third parameter of the at least one parameter; or updating the at least part of the quantized implicit representation by adding at least a fourth parameter of the at least one parameter.

[0100] Clause 4. A method according to any of clauses 2-3, wherein the prediction is an average, or the difference is included in a quantized residual latent representation of the visual data.

[0101] Clause 5. The method of any one of clauses 2-4, wherein obtaining the at least a portion of the quantized implicit representation comprises: performing an entropy decoding process on the bitstream to obtain the at least a portion of the quantized implicit representation.

[0102] Clause 6. The method of clause 5, further comprising: generating the prediction by using a first model.

[0103] Clause 7. A method according to any one of clauses 2-4, wherein obtaining the at least part of the quantized implicit representation comprises: obtaining the difference by performing entropy decoding processing on the bitstream; generating the prediction by using a first model; and generating the at least part of the quantized implicit representation based on the prediction and the difference.

[0104] Clause 8. A method according to any of clauses 6-7, wherein generating the prediction comprises: generating a prediction of samples in the at least a portion of the quantized implicit representation based on at least one reconstructed sample of the quantized implicit representation by using the first model.

[0105] Clause 9. The method of any one of clauses 6-8, wherein the first model is a predictive model.

[0106] Clause 10. The method of any one of clauses 6-9, wherein the first model is autoregressive.

[0107] Clause 11. A method according to any of clauses 6-10, wherein the first model comprises a context sub-network or a context model sub-network.

[0108] Clause 12. A method according to any of clauses 2-11, wherein the at least a portion of the quantized implicit representation includes all samples of the quantized implicit representation, and the intermediate representation corresponds to a result of the update.

[0109] Clause 13. The method according to any one of clauses 2-11 further comprises: updating another part of the quantized implicit representation based on at least one additional parameter different from the at least one parameter, the other part being different from the at least one part of the quantized implicit representation.

[0110] Clause 14. The method of clause 1, wherein obtaining the intermediate representation comprises generating at least a portion of the intermediate representation based on the prediction and the difference.

[0111] Clause 15. A method according to clause 14, wherein generating at least a portion of the intermediate representation comprises: generating at least a portion of the intermediate representation by summing the product of the prediction and a first parameter among the at least one parameter, and the product of the difference and a second parameter among the at least one parameter, or generating at least a portion of the intermediate representation by summing the product of the prediction and the first parameter, the product of the difference and the second parameter, and a third parameter among the at least one parameter.

[0112] Clause 16. The method of any of clauses 14-15, wherein at least one of the prediction or the difference is generated using a first model.

[0113] Clause 17. The method of any one of clauses 6-8 and 16, wherein the first model is an estimated model.

[0114] Clause 18. A method according to any one of clauses 6-8 and 16-17, wherein the first model comprises a sub-network based on a neural network, or the input of the first model comprises the bitstream.

[0115] Clause 19. The method of any of clauses 6-8 and 16-17, wherein the first model comprises a first subnetwork for generating the predictions and a second subnetwork for generating statistics.

[0116] Clause 20. The method of Clause 19, wherein the first sub-network is a super decoder sub-network and the second sub-network is a superscale decoder sub-network.

[0117] Clause 21. The method of any of clauses 1-20, further comprising: determining the at least a portion of the quantized implicit representation from the quantized implicit representation.

[0118] Clause 22. A method according to Clause 21, wherein determining at least a portion of the quantized implicit representation from the quantized implicit representation includes: determining whether the at least a portion of the quantized implicit representation includes a sample of the quantized implicit representation based on at least one of the following: a comparison between a first threshold and a statistical value corresponding to the sample, a comparison between a second threshold and a value determined based on the statistical value, or a comparison between a third threshold and an index of the sample.

[0119] Clause 23. A method according to Clause 21, wherein determining the at least a portion of the quantized implicit representation from the quantized implicit representation includes: determining whether the at least a portion of the quantized implicit representation includes samples in the block of the quantized implicit representation based on at least one of the following: a comparison between a fourth threshold and a statistical value corresponding to each of the samples, a comparison between a fifth threshold and a metric determined based on the statistical value corresponding to the sample, or a comparison between a sixth threshold and an index of each of the samples.

[0120] Clause 24. The method of clause 23, wherein the metric is an average, a minimum, or a maximum.

[0121] Clause 25. A method according to any one of clauses 22-24, wherein the index of a sample indicates one of the following: a channel number of the sample, a feature map identifier of the sample, or a spatial coordinate of the sample.

[0122] Clause 26. A method according to any one of clauses 22-25, wherein at least one of the following thresholds or an indication of at least one of the following is indicated in the bitstream: the first threshold, the second threshold, the third threshold, the fourth threshold, the fifth threshold, or the sixth threshold.

[0123] Clause 27. The method of any one of clauses 19-26, wherein the statistical value is a variance.

[0124] Clause 28. The method according to any one of clauses 19-27, wherein the statistical value is a variance of a Gaussian probability, or the statistical value is obtained based on the bitstream by using the second sub-network.

[0125] Clause 29. The method of any of clauses 1-28, wherein the at least one parameter or an indication of the at least one parameter is included in the bitstream.

[0126] Clause 30. The method of any one of clauses 1-29, wherein the at least one parameter is a scalar value different from zero, or a vector.

[0127] Clause 31. The method of any of clauses 1-30, wherein the at least one parameter is determined based on a quality metric.

[0128] Clause 32. The method of clause 31, wherein the quality metric comprises at least one of: mean square error, structural similarity (SSIM), or multi-scale structural similarity (MS-SSIM).

[0129] Clause 33. A method according to any of clauses 1-32, wherein the at least a portion of the quantized implicit representation comprises one or more samples of the quantized implicit representation.

[0130] Clause 34. A method according to any of clauses 1-33, wherein the synthetic transformation is performed using a neural network based sub-network, or the first neural network is used to perform an analytical transformation on the visual data.

[0131] Clause 35. A method according to any of clauses 1-34, wherein at least one of the following is indicated in the bitstream: information about whether to apply the method, or information about how to apply the method.

[0132] Clause 36. A method according to any of clauses 1-34, wherein at least one of the following depends on the color format and / or color components of the visual data: information about whether to apply the method, or information about how to apply the method.

[0133] Clause 37. The method of any of clauses 1-36, wherein the values ​​included in the bitstream are encoded at one of: a sequence level, a picture level, a slice level, or a block level.

[0134] Clause 38. The method of any of clauses 1-37, wherein values ​​included in the bitstream are binarized prior to being encoded or decoded.

[0135] Clause 39. The method of any of clauses 1-38, wherein values ​​included in the bitstream are encoded using at least one arithmetic coding context.

[0136] Clause 40. The method of any one of clauses 1-39, wherein the visual data comprises a picture or image of a video.

[0137] Clause 41. The method of any of Clauses 1-40, wherein the converting comprises encoding the visual data into the bitstream.

[0138] Clause 42. The method of any of clauses 1-40, wherein the converting comprises decoding the visual data from the bitstream.

[0139] Clause 43. An apparatus for visual data processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of clauses 1-42.

[0140] Clause 44. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method of any of clauses 1-42.

[0141] Item 45. A non-transitory computer-readable recording medium storing a bitstream of visual data, the bitstream being generated by a method performed by a visual data processing device, wherein the method comprises: obtaining an intermediate representation of the visual data, the intermediate representation being different from a quantized implicit representation of the visual data and being generated based on at least one of: at least one parameter, at least a portion of the quantized implicit representation, a prediction of the at least a portion of the quantized implicit representation, or a difference between the prediction and the at least a portion of the quantized implicit representation; and generating the bitstream based on a synthetic transformation of the intermediate representation, wherein the quantized implicit representation is generated based on applying a first neural network to the visual data.

[0142] Clause 46. A method for storing a bitstream of visual data, comprising: obtaining an intermediate representation of the visual data, the intermediate representation being different from a quantized implicit representation of the visual data and being generated based on at least one of: at least one parameter, at least a portion of the quantized implicit representation, a prediction of the at least a portion of the quantized implicit representation, or a difference between the prediction and the at least a portion of the quantized implicit representation; generating the bitstream based on a synthetic transformation of the intermediate representation, and storing the bitstream in a non-transitory computer-readable recording medium, wherein the quantized implicit representation is generated based on applying a first neural network to the visual data. Example Device

[0143] Fig.18 A block diagram of a computing device 1800 is shown in which various embodiments of the present disclosure may be implemented. The computing device 1800 may be implemented as or included in the source device 110 (or visual data encoder 114) or the destination device 120 (or visual data decoder 124).

[0144] It should be understood that Fig.18 The computing device 1800 shown in FIG. 1 is for illustrative purposes only and is not intended to in any way imply any limitation on the functionality and scope of the embodiments of the present disclosure.

[0145] like Fig.18 As shown, computing device 1800 includes a general computing device 1800. Computing device 1800 may include at least one or more processors or processing units 1810, memory 1820, storage unit 1830, one or more communication units 1840, one or more input devices 1850, and one or more output devices 1860.

[0146] In some embodiments, computing device 1800 can be implemented as any user terminal or server terminal with computing power. The server terminal can be a server provided by a service provider, a large computing device, etc. The user terminal can be, for example, any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet computer, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device, or any combination thereof, and includes the accessories and peripherals of these devices, or any combination thereof. It is conceivable that computing device 1800 can support any type of interface to the user (such as a "wearable" circuit device, etc.).

[0147] The processing unit 1810 may be a physical processor or a virtual processor and may implement various processes based on a program stored in the memory 1820. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the computing device 1800. The processing unit 1810 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0148] The computing device 1800 typically includes various computer storage media. Such media can be any media accessible by the computing device 1800, including but not limited to volatile media and non-volatile media, or removable media and non-removable media. The memory 1820 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM) or flash memory) or any combination thereof. The storage unit 1830 can be any removable or non-removable medium, and can include machine-readable media, such as a memory, a flash drive, a disk, or other media that can be used to store information and / or visual data and can be accessed in the computing device 1800.

[0149] The computing device 1800 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Fig.18It is not shown in the figure, but a disk drive for reading from and / or writing to a removable non-volatile disk and an optical drive for reading from and / or writing to a removable non-volatile optical disk may be provided. In this case, each drive may be connected to the bus (not shown) via one or more visual data medium interfaces.

[0150] The communication unit 1840 communicates with another computing device via a communication medium. In addition, the functions of the components in the computing device 1800 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 1800 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general network nodes.

[0151] The input device 1850 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. The output device 1860 may be one or more of various output devices, such as a display, a speaker, a printer, etc. With the help of the communication unit 1840, the computing device 1800 may also communicate with one or more external devices (not shown), such as storage devices and display devices, and the computing device 1800 may also communicate with one or more devices that enable a user to interact with the computing device 1800, or any device that enables the computing device 1800 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.), if necessary. Such communication may be performed via an input / output (I / O) interface (not shown).

[0152] In some embodiments, some or all components of computing device 1800 may also be arranged in a cloud computing architecture rather than being integrated in a single device. In a cloud computing architecture, components may be provided remotely and work together to implement the functions described in the present disclosure. In some embodiments, cloud computing provides computing, software, visual data access and storage services, which will not require the end user to know the physical location or configuration of the system or hardware that provides these services. In various embodiments, cloud computing provides services via a wide area network (e.g., the Internet) using a suitable protocol. For example, a cloud computing provider provides applications via a wide area network, which can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding visual data may be stored on a remote server. The computing resources in a cloud computing environment may be merged or distributed at the location of a remote visual data center. Cloud computing infrastructures may provide services through a shared visual data center, although they appear as a single access point for users. Therefore, cloud computing architectures may be used to provide components and functions described herein from a service provider at a remote location. Alternatively, they may be provided by a conventional server, or installed directly or otherwise on a client device.

[0153] The computing device 1800 can be used to implement the visual data encoding / decoding in the embodiments of the present disclosure, and the memory 1820 can include one or more visual data encoding / decoding modules 1825 having one or more program instructions. These modules can be accessed and executed by the processing unit 1810 to perform the functions of the various embodiments described herein.

[0154] In an example embodiment where visual data encoding is performed, input device 1850 may receive visual data as input 1850 to be encoded. The visual data may be processed by, for example, visual data encoding / decoding module 1825 to generate an encoded bitstream. The encoded bitstream may be provided as output 1880 via output device 1860.

[0155] In an example embodiment where visual data decoding is performed, input device 1850 may receive an encoded bitstream as input 1850. The encoded bitstream may be processed by, for example, visual data codec module 1825 to generate decoded visual data. The decoded visual data may be provided as output 1880 via output device 1860.

[0156] Although the present disclosure has been specifically shown and described with reference to the preferred embodiments of the present disclosure, it will be understood by those skilled in the art that various changes can be made in form and detail without departing from the spirit and scope of the present application as defined by the appended claims. These changes are intended to be encompassed by the scope of the present application. Therefore, the foregoing description of the embodiments of the present application is not intended to be limiting.

Claims

1. A method for visual data processing, comprising: For conversion between visual data and a bitstream of the visual data, an intermediate representation of the visual data is obtained, the intermediate representation being different from a quantized implicit representation of the visual data and being generated based on at least one of the following: At least one parameter, at least a portion of the quantized implicit representation, a prediction of said at least a portion of said quantized implicit representation, or a difference between the prediction and the at least a portion of the quantized implicit representation; and for said conversion, performing a composition transform on said intermediate representation, Wherein the quantized latent representation is generated based on applying a first neural network to the visual data.

2. The method according to claim 1, wherein obtaining the intermediate representation comprises: obtaining the at least a portion of the quantized implicit representation; as well as Based on at least one of the at least one parameter, the prediction, or the difference, the at least a portion of the quantized implicit representation is updated to obtain at least a portion of the intermediate representation.

3. The method of claim 2, wherein updating the at least a portion of the quantized implicit representation comprises at least one of: scaling the at least a portion of the quantized implicit representation using a first parameter of the at least one parameter; updating the at least a portion of the quantized implicit representation based on a product of the prediction and a second parameter of the at least one parameter and / or a product of the difference and a third parameter of the at least one parameter; or The at least a portion of the quantized implicit representation is updated by adding at least a fourth parameter of the at least one parameter.

4. A method according to any one of claims 2-3, wherein the prediction is an average value, or the difference is included in a quantized residual latent representation of the visual data.

5. The method according to any one of claims 2 to 4, wherein obtaining the at least a portion of the quantized implicit representation comprises: An entropy decoding process is performed on the bitstream to obtain the at least a portion of the quantized implicit representation.

6. The method according to claim 5, further comprising: The prediction is generated by using a first model.

7. The method according to any one of claims 2 to 4, wherein obtaining the at least a portion of the quantized implicit representation comprises: Obtaining the difference by performing entropy decoding processing on the bit stream; generating the prediction by using a first model; as well as Based on the prediction and the difference, the at least a portion of the quantized implicit representation is generated.

8. The method of any one of claims 6-7, wherein generating the prediction comprises: A prediction of samples in the at least a portion of the quantized implicit representation is generated based on at least one reconstructed sample of the quantized implicit representation using the first model.

9. The method according to any one of claims 6-8, wherein the first model is a predictive model.

10. The method according to any one of claims 6-9, wherein the first model is autoregressive.

11. The method according to any one of claims 6-10, wherein the first model comprises a context sub-network or a context model sub-network.

12. The method according to any one of claims 2-11, wherein the at least part of the quantized latent representation comprises all samples of the quantized latent representation, and the intermediate representation corresponds to a result of the update.

13. The method according to any one of claims 2 to 11, further comprising: Another portion of the quantized implicit representation is updated based on at least one further parameter different from the at least one parameter, the another portion being different from the at least one portion of the quantized implicit representation.

14. The method of claim 1, wherein obtaining the intermediate representation comprises: At least a portion of the intermediate representation is generated based on the prediction and the difference.

15. The method of claim 14, wherein generating the at least a portion of the intermediate representation comprises: The at least a portion of the intermediate representation is generated by summing: a product of the prediction and a first parameter of the at least one parameter, and The at least one portion of the intermediate representation is generated by multiplying the difference by a second parameter of the at least one parameter, or by summing: the product of the prediction and the first parameter, the product of the difference and the second parameter, and A third parameter of the at least one parameter.

16. The method of any one of claims 14-15, wherein at least one of the prediction or the difference is generated using a first model.

17. The method according to any one of claims 6-8 and 16, wherein the first model is an estimation model.

18. The method according to any one of claims 6-8 and 16-17, wherein the first model comprises a sub-network based on a neural network, or an input of the first model comprises the bitstream.

19. The method of any one of claims 6-8 and 16-17, wherein the first model comprises a first subnetwork for generating the predictions and a second subnetwork for generating statistics.

20. The method of claim 19, wherein the first sub-network is a super decoder sub-network and the second sub-network is a superscale decoder sub-network.

21. The method according to any one of claims 1 to 20, further comprising: The at least a portion of the quantized implicit representation is determined from the quantized implicit representation.

22. The method of claim 21 , wherein determining the at least a portion of the quantized implicit representation from the quantized implicit representation comprises: Determining whether the at least a portion of the quantized implicit representation includes a sample of the quantized implicit representation is based on at least one of: a comparison between a first threshold and a statistical value corresponding to the sample, a comparison between a second threshold and a value determined based on the statistical value, or A comparison is made between a third threshold and the index of the sample.

23. The method of claim 21 , wherein determining the at least a portion of the quantized implicit representation from the quantized implicit representation comprises: determining whether the at least a portion of the quantized implicit representation includes samples in a block of the quantized implicit representation based on at least one of: a comparison between a fourth threshold and a statistical value corresponding to each of the samples, a comparison between a fifth threshold and a metric determined based on a statistical value corresponding to the sample, or A comparison between a sixth threshold and the index of each of the samples.

24. The method of claim 23, wherein the metric is an average, a minimum, or a maximum.

25. The method of any one of claims 22-24, wherein the index of the sample indicates one of: The channel number of the sample, the feature map identifier of the sample, or The spatial coordinates of the sample.

26. The method of any one of claims 22-25, wherein at least one of the following thresholds or an indication of at least one of the following is indicated in the bitstream: The first threshold, The second threshold, The third threshold, The fourth threshold, the fifth threshold, or The sixth threshold.

27. The method according to any one of claims 19-26, wherein the statistical value is the variance.

28. The method according to any one of claims 19 to 27, wherein the statistical value is a variance of a Gaussian probability, or the statistical value is obtained by using the second sub-network based on the bit stream.

29. The method according to any one of claims 1-28, wherein the at least one parameter or an indication of the at least one parameter is included in the bitstream.

30. The method of any one of claims 1-29, wherein the at least one parameter is a scalar value or a vector different from zero.

31. The method according to any one of claims 1-30, wherein the at least one parameter is determined based on a quality metric.

32. The method of claim 31 , wherein the quality metric comprises at least one of: Mean square error, Structural Similarity (SSIM), or Multi-Scale Structural Similarity (MS-SSIM).

33. The method of any one of claims 1-32, wherein the at least a portion of the quantized latent representation comprises one or more samples of the quantized latent representation.

34. A method according to any one of claims 1 to 33, wherein the synthetic transformation is performed by using a neural network based sub-network, or The first neural network is used to perform an analytical transformation on the visual data.

35. The method of any one of claims 1-34, wherein at least one of the following is indicated in the bitstream: information on whether the method was applied, or Information on how to apply the method.

36. The method according to any one of claims 1 to 34, wherein at least one of the following depends on the color format and / or color components of the visual data: information on whether the method was applied, or Information on how to apply the method.

37. The method of any one of claims 1-36, wherein a value included in the bitstream is encoded or decoded at one of: Sequence level, Picture level, Stripe level, or Block level.

38. The method according to any one of claims 1-37, wherein the values ​​included in the bitstream are binarized before being encoded and decoded.

39. The method according to any one of claims 1-38, wherein the values ​​included in the bitstream are encoded using at least one arithmetic coding context.

40. The method of any one of claims 1-39, wherein the visual data comprises a picture or image of a video.

41. The method of any one of claims 1-40, wherein the converting comprises encoding the visual data into the bitstream.

42. The method of any one of claims 1-40, wherein the converting comprises decoding the visual data from the bitstream.

43. An apparatus for visual data processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1-42.

44. A non-transitory computer-readable storage medium storing instructions, the instructions causing a processor to execute the method according to any one of claims 1-42.

45. A non-transitory computer-readable recording medium storing a bit stream of visual data, the bit stream being generated by a method performed by a visual data processing device, wherein the method comprises: Obtaining an intermediate representation of the visual data, the intermediate representation being different from a quantized latent representation of the visual data and being generated based on at least one of: At least one parameter, at least a portion of the quantized implicit representation, a prediction of said at least a portion of said quantized implicit representation, or a difference between the prediction and the at least a portion of the quantized implicit representation; and generating the bitstream based on a synthetic transformation of the intermediate representation, Wherein the quantized latent representation is generated based on applying a first neural network to the visual data.

46. ​​A method for storing a bitstream of visual data, comprising: Obtaining an intermediate representation of the visual data, the intermediate representation being different from a quantized latent representation of the visual data and being generated based on at least one of: At least one parameter, at least a portion of the quantized implicit representation, a prediction of said at least a portion of said quantized implicit representation, or a difference between the prediction and the at least a portion of the quantized implicit representation; generating the bitstream based on a synthetic transformation of the intermediate representation, and storing the bitstream in a non-transitory computer-readable recording medium, Wherein the quantized latent representation is generated based on applying a first neural network to the visual data.

Images