Method, device and medium for visual data processing

Abstract

Embodiments of the present disclosure provide a solution for visual data processing. A method for visual data processing is presented. The method comprises: for a conversion between visual data and a bitstream of visual data, obtaining statistical information associated with a first representation of the visual data, the first representation being generated based on applying a first neural network to the visual data; determining, based on the statistical information, at least one sample from a second representation of the visual data, the value of the at least one sample being not in the bitstream, the second representation being obtained by quantizing the first representation; and performing a conversion based on the determination.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to visual data processing technology, and more specifically, to neural network-based visual data encoding and decoding. Background Art

[0002] The past decade has witnessed the rapid development of deep learning in various fields, especially in computer vision and image processing. Neural networks were originally invented using interdisciplinary research in neuroscience and mathematics. It has shown strong capabilities in nonlinear transformations and classification. Neural network-based image / video compression technology has made significant progress during the past five years. It is reported that the latest neural network-based image compression algorithms achieve rate-distortion (RD) performance comparable to that of the general video codec (VVC). As the performance of neural images continues to improve, neural network-based video compression has become an actively developing research area. However, it is generally desired to further improve the codec efficiency of neural network-based image / video codecs. Summary of the invention

[0003] Embodiments of the present disclosure provide a solution for visual data processing.

[0004] In a first aspect, a method for visual data processing is provided. The method includes: for conversion between visual data and a bitstream of the visual data, obtaining statistical information associated with a first representation of the visual data, the first representation being generated based on applying a first neural network to the visual data; determining at least one sample from a second representation of the visual data based on the statistical information, the value of the at least one sample not being in the bitstream, the second representation being obtained by quantizing the first representation; and performing conversion based on the determination.

[0005] According to the method of the first aspect of the present disclosure, the value of at least one sample of the quantization of the representation of the visual data is not included in the bitstream. Therefore, the proposed method enables at least a portion of the quantization to be skipped during the entropy coding and decoding process. In this way, the proposed method can advantageously improve the coding and decoding speed and coding and decoding efficiency.

[0006] In a second aspect, a device for visual data processing is provided. The device includes a processor and a non-volatile memory having instructions thereon. The instructions, when executed by the processor, cause the processor to perform the method according to the first aspect of the present disclosure.

[0007] In a third aspect, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions, which enable a processor to execute the method according to the first aspect of the present disclosure.

[0008] In a fourth aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of visual data, the bitstream being generated by a method performed by a visual data processing device. The method includes: for conversion between visual data and a bitstream of visual data, obtaining statistical information associated with a first representation of the visual data, the first representation being generated based on applying a first neural network to the visual data; determining at least one sample from a second representation of the visual data based on the statistical information, the value of the at least one sample not being in the bitstream, the second representation being obtained by quantizing the first representation; and generating a bitstream based on the determination.

[0009] In a fifth aspect, a method for storing a bitstream of visual data is provided. The method includes: for conversion between visual data and a bitstream of visual data, obtaining statistical information associated with a first representation of the visual data, the first representation being generated based on applying a first neural network to the visual data; determining at least one sample from a second representation of the visual data based on the statistical information, the value of the at least one sample not being in the bitstream, the second representation being obtained by quantizing the first representation; generating a bitstream based on the determination; and storing the bitstream in a non-transitory computer-readable recording medium.

[0010] This disclosure is provided to introduce a selection of concepts in a simplified form, which are further described below in the Detailed Description. This disclosure is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become more apparent through the following detailed description with reference to the accompanying drawings.In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0012] Figure 1 A block diagram illustrating an example data encoding and decoding system according to some embodiments of the present disclosure is shown;

[0013] Figure 2 A typical transform coding scheme is shown;

[0014] Figure 3 An image from the Kodak dataset and different representations of the image are shown;

[0015] Figure 4 The network architecture of the autoencoder implementing the hyper-prior model is shown;

[0016] Figure 5 A block diagram of the combined model is shown;

[0017] Figure 6The encoding process of the combined model is shown;

[0018] Figure 7 The decoding process of the combined model is shown;

[0019] Figure 8 Two possible implementations of the entropy encoding and decoding and quantization process are shown;

[0020] Fig. 9 The corresponding decoder implementation for the encoder implementation is shown;

[0021] Fig.10 Two possible implementations of an encoder according to an embodiment of the present disclosure are shown;

[0022] Fig.11 Two possible implementations of a decoder according to an embodiment of the present disclosure are shown;

[0023] Fig.12 Two possible probability distributions according to embodiments of the present disclosure are shown;

[0024] Fig.13 An example visual data encoding process according to some embodiments of the present disclosure is shown;

[0025] Fig.14 shows an example visual data decoding process according to some embodiments of the present disclosure;

[0026] Fig.15 A flowchart showing a method for visual data processing according to some embodiments of the present disclosure; and

[0027] Fig.16 A block diagram of a computing device is shown in which various embodiments of the present disclosure may be implemented.

[0028] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. DETAILED DESCRIPTION

[0029] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are only for illustrative purposes, and contribute to those skilled in the art to understand and implement the present disclosure, without implying any limitation to the scope of the present disclosure. The disclosure described herein can be realized in various ways different from the modes described below.

[0030] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0031] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example embodiment, it is submitted that it is within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in conjunction with other embodiments, whether or not explicitly described.

[0032] It should be understood that although the terms "first" and "second" etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the example embodiments, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0033] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the terms "include", "including", "have", "have", "include" and / or "include" specify the presence of the features, elements and / or components, etc. when used herein, but do not exclude the presence or addition of one or more other features, elements, components and / or combinations thereof. Example Environment

[0034] Figure 1 1 is a block diagram illustrating an example visual data encoding and decoding system 100 that may utilize the techniques of the present disclosure. As shown, the visual data encoding and decoding system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a visual data encoding device, and the destination device 120 may also be referred to as a visual data decoding device. In operation, the source device 110 may be configured to generate encoded visual data, and the destination device 120 may be configured to decode the encoded visual data generated by the source device 110. The source device 110 may include a visual data source 112, a visual data encoder 114, and an input / output (I / O) interface 116.

[0035] Visual data source 112 may include a source such as a visual data capture device. Examples of a visual data capture device include, but are not limited to, an interface for receiving visual data from a visual data provider, a computer graphics system for generating visual data, and / or combinations thereof.

[0036] The visual data may include one or more images or one or more pictures of a video. The visual data encoder 114 encodes the visual data from the visual data source 112 to generate a bitstream. The bitstream may include a bit sequence that forms a coded representation of the visual data. The bitstream may include a coded picture and associated visual data. The coded picture is a coded representation of a picture. The associated visual data may include a sequence parameter set, a picture parameter set, and other grammatical structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The coded visual data may be directly transmitted to the destination device 120 via the network 130A via the I / O interface 116. The coded visual data may also be stored on a storage medium / server 130B for access by the destination device 120.

[0037] Destination device 120 may include an I / O interface 126, a visual data decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain the encoded visual data from source device 110 or storage medium / server 130B. The visual data decoder 124 may decode the encoded visual data. The display device 122 may display the decoded visual data to the user. The display device 122 may be integrated with the destination device 120, or may be external to the destination device 120, which is configured to be connected to an external display device interface.

[0038] The visual data encoder 114 and the visual data decoder 124 may operate according to a visual data codec standard, such as a video codec standard or a still image codec standard, as well as other current and / or further standards.

[0039] Some exemplary embodiments of the present disclosure will be described in detail below. It should be understood that section titles are used in this document to promote ease of understanding, and the embodiments disclosed in a section are not limited to that section. In addition, although certain embodiments are described with reference to general video codecs or other specific visual data codecs, the disclosed technology is also applicable to other codec technologies. In addition, although some embodiments describe the coding and decoding steps in detail, it should be understood that the corresponding step decoding of the undo coding and decoding will be implemented by the decoder. In addition, the term visual data processing includes visual data coding or compression, visual data decoding or decompression, and visual data transcoding, in which visual data is represented from one compression format to another compression format or with different compression bit rates. 2.1. Image / Video Compression Image / video compression generally refers to computing techniques that compress images / videos into binary codes for easy storage and transmission. The binary codes may or may not support lossless reconstruction of the original image / video, known as lossless compression and lossy compression. Most research is devoted to lossy compression, since lossless reconstruction is not required in most scenarios. Usually, the performance of image / video compression algorithms is evaluated from two aspects, namely compression ratio and reconstruction quality. The compression ratio is directly related to the number of binary codes, the worse it is; the reconstruction quality is measured by comparing the reconstructed image / video with the original image / video, the better it is. Image / video compression technology can be divided into two branches, classical video codec methods and neural network-based video compression methods. Classical video coding schemes use transform-based solutions, in which researchers exploit the statistical dependencies of these latent variables (e.g., DCT or wavelet coefficients) in the quantization domain through carefully hand-written entropy codes. There are two forms of neural network-based video compression: neural network-based codec tools and end-to-end neural network-based video compression. The former is embedded into existing classical video codecs as codec tools and is only used as part of the framework, while the latter is a separate framework developed based on neural networks without relying on classical video codecs. Over the past three decades, a series of classic video codec standards have been developed to accommodate the increased visual content. The international standards organization ISO / IEC has two expert groups, the Joint Photographic Experts Group (JPEG) and the Moving Picture Experts Group (MPEG), and ITU-T also has its own Video Codec Experts Group (VCEG), which is used for the standardization of image / video codec technologies. Influential video codec standards released by these organizations include JPEG, JPEG 2000, H.262, H.264 / AVC, and H.265 / HEVC. After H.265 / HEVC, the Joint Video Experts Group (JVET) formed by MPEG and VCEG has been working on a new video codec standard, the Versatile Video Codec (VVC). The first version of VVC was released in July 2020. It is reported that VVC achieves an average bit rate reduction of 50% over HEVC at the same visual quality. Neural network-based image / video compression is not a new invention, as there are many researchers working on image codecs based on neural networks. However, the network architecture is relatively shallow and the performance is not satisfactory. Benefiting from the support of large amounts of data and powerful computing resources, neural network-based methods are better utilized in various applications. Currently, neural network-based image / video compression has shown promising improvements, proving its feasibility. However, the technology is still far from mature and many challenges need to be addressed. Neural Networks A neural network, also known as an artificial neural network (ANN), is a computational model used in machine learning techniques that is typically composed of multiple processing layers, and each layer is composed of multiple simple but nonlinear basic computational units. One benefit of such deep networks is believed to be the ability to process data with multiple layers of abstraction and transform the data into different types of representations. Note that these representations are not manually designed; instead, the deep network including the processing layers is learned from massive amounts of data through general machine learning programs. Deep learning eliminates the need for manual representations and is therefore believed to be useful, particularly for processing originally unstructured data, such as acoustic and visual signals, which has been a long-standing problem in the field of artificial intelligence. 2.3. Neural Networks for Image Compression Existing neural networks for image methods can be divided into two categories, namely pixel probability modeling and autoencoders. The former belongs to the predictive encoding and decoding strategy, while the latter is a transformation-based solution. Sometimes, these two methods are combined in the literature. 2.3.1. Pixel Probabilistic Modeling According to Shannon's information theory, the best method for lossless coding can achieve the minimum coding rate -log2p(x), where p(x) is the symbol probability. Many lossless coding methods have been developed in the literature, among which arithmetic coding is considered to be one of the best methods. Given a probability distribution p(x), arithmetic coding ensures that the coding rate is as close to its theoretical limit -log2p(x) as possible without considering rounding errors. Therefore, the remaining question is how to determine the probability, which is very challenging for natural images / videos due to the reduction in dimensionality. Following the predictive coding strategy, one way to model p(x) is to predict pixel probabilities one by one in raster scan order based on previous observations, where x is an image. p(x)=p(x1)p(x2|x1)…p(x i |x1,…,x i-1 )…p(x m×n |x1,…,x m×n-1 ) (1) Where m and n are the height and width of the image, respectively. The previous observation is also called the context of the current pixel. When the image is large, it may be difficult to estimate the conditional probability, so a simplified approach is to limit the scope of its context. p(x)=p(x1)p(x2|x1)…p(x i |x i-k ,…,x i-1 )…p(x m×n |x m×n-k ,…,x m×n-1 ) (2)where k is a predefined constant that controls the scope of the context. It should be noted that the conditions may also take into account sample values ​​of other color components. For example, when encoding and decoding RGB color components, the R sample depends on previously encoded and decoded pixels (including R / G / B samples), the current G sample may be encoded and decoded based on the previously encoded and decoded pixels and the current R sample, and for encoding and decoding the current B sample, the previously encoded and decoded pixels and the current R and G samples may also be considered. Neural networks were initially introduced for computer vision tasks and have been shown to be effective in regression and classification problems. Therefore, it has been proposed to use neural networks to estimate a given its context x1,x2,…,x i-1 The probability p(x i ). In [7], for a binary image, that is, x i ∈{-1,+1}, pixel probability is proposed. The Neural Autoregressive Distribution Estimator (NADF) is designed for pixel probability modeling and is a feedforward network with a single hidden layer. Similar work has been presented in existing designs, where the feedforward network also has connections that skip the hidden layer and parameters are also shared p(x i |x1,…,x i-1 ). Experiments are performed on the binarized MNIST dataset. In the existing design, NADF is extended to the real-valued model RNADE, where the probabilities are derived using a mixture of Gaussians. Their feed-forward network also has a single hidden layer, but the hidden layer is rescaled to avoid saturation and uses rectified linear units (ReLU) instead of sigmoids. In the existing design, NADF and RNADE are improved by rearranging the order of pixels and utilizing deeper neural networks. Designing advanced neural networks plays an important role in improving pixel probability modeling. In existing designs, multidimensional long short-term memory (LSTM) is proposed, which is used for probability modeling together with conditional Gaussian scaled mixture. LSTM is a special kind of recurrent neural network (RNN) and is proven to be good at modeling sequential data. Spatial variants of LSTM are used for images in existing designs. Several different neural networks are studied, including RNN and CNN, namely PixelRNN and PixelCNN. In PixelRNN, two variants of LSTM are proposed, called row LSTM and diagonal BiLSTM, of which the latter is specifically designed for images. PixelRNN contains residual connections to help train deep neural networks with up to 12 layers. In PixelCNN, masked convolution is used to adapt to the shape of the context. Compared with previous work, PixelRNN and PixelCNN are more specialized for natural images: they treat pixels as discrete values ​​(e.g., 0, 1, ..., 255) and predict a multinomial distribution of discrete values; they process color images in RGB color space; they work well on the large-scale image dataset ImageNet. In the existing design, Gated PixelCNN is proposed to improve PixelCNN and achieve comparable performance to PixelRNN, but with much lower complexity. In the existing design, PixelCNN++ has the following improvements on PixelCNN: discretized logistic mixture likelihood is used instead of multinomial distribution of 256 channels; downsampling is used to capture structure at multiple resolutions; additional shortcut connections are introduced to speed up training; dropout is used for regularization; RGB is combined for one pixel. In the existing design, PixelSNAIL is proposed, in which causal convolution is combined with self-attention. Most of the above methods directly model the probability distribution in the pixel domain. Some researchers have also tried to model the probability distribution as a conditional distribution based on explicit or implicit representations. That is, it can be estimated Where h is an additional condition and p(x)=p(h)p(x|h), which means that the modeling is divided into an unconditional part and a conditional part. The additional condition can be image label information or a high-level representation. 2.3.2. Autoencoder Autoencoders are derived from the well-known work proposed by Hinton and Salakhutdinov. The method is trained for dimensionality reduction and consists of two parts: encoding and decoding. The encoding part converts the high-dimensional input signal into a low-dimensional representation, usually with a reduced spatial size but with a larger number of channels. The decoding part tries to recover the high-dimensional input from the low-dimensional representation. Autoencoders enable automatic learning of representations and eliminate the need for hand-crafted features, which is also considered one of the most important advantages of neural networks. Figure 2 A typical transform coding scheme is shown. The original image x is analyzed by the network g a Transform to achieve the implicit representation y. The implicit representation y is quantized and compressed into bits. The number of bits R is used to measure the encoding and decoding rate. Then, the synthetic network g s The quantized implicit representation Perform inverse transform to obtain the reconstructed image By using the function g p Transform x and x to compute the distortion in perceptual space. It is intuitive to apply autoencoder networks to lossy compression. This only requires encoding the learned latent representations from a trained neural network. However, adapting the autoencoder to images is not simple because the original autoencoder is not optimized for compression, so directly using the trained autoencoder is not efficient. Additionally, there are other major challenges: First, the low-dimensional representation should be quantized before encoding, but quantization is not differentiable, which is required in backpropagation when training neural networks. Second, the goals in the compression scenario are different because both distortion and rate need to be considered. Estimation of the bitrate is challenging. Third, practical image coding and decoding schemes need to support variable bitrate, scalability, encoding / decoding speed, and interoperability. To address these challenges, many researchers have actively contributed to this field. Prototype autoencoder for image compression in Figure 2 It can be regarded as a transform coding strategy. a (x) Transform the original image x, where y is the latent representation that will be quantized and decoded. The synthesis network will inverse transform the quantized latent representation to obtain the reconstructed image The training framework is based on the rate-distortion loss function, i.e., where D is x and The distortion between R is the quantized representation The bit rate is calculated or estimated, and λ is the Lagrange multiplier. It should be noted that D can be calculated in the pixel domain or the perceptual domain. All existing research works follow this prototype, and the difference may only be the network structure or loss function. In terms of network structure, RNN and CNN are the most widely used architectures. In the RNN-related category, Toderici et al. proposed a general framework for variable bitrate images using RNN. They used binary quantization to generate codes and did not consider the bitrate during training. The framework actually provides scalable encoding and decoding capabilities, and RNNs with convolutional and deconvolutional layers are reported to perform well. Then, Toderici et al. proposed an improved version by upgrading the encoder using a neural network similar to PixelRNN to compress the binary code. It is reported that the method performs better than JPEG on the Kodak image dataset using the MS-SSIM evaluation metric. Johnston et al. further improved the RNN-based solution by introducing hidden state preprocessing. In addition, an SSIM weighted loss function is designed and a spatially adaptive bitrate mechanism is enabled. Their results on the Kodak image dataset are better than BPG using MS-SSIM as the evaluation metric. Covell et al. support spatially adaptive bitrate by training a fault-tolerant stop encoding and decoding RNN. Ballé et al. proposed a general framework for rate-distortion optimized images. Integer codes are generated using multivariate quantization, and the code rate is considered during training, i.e., the loss is a joint rate-distortion cost, which can be either mean squared error (MSE) or other forms. They add random uniform noise during training to motivate quantization, and use the differential entropy of the noise code as a proxy for the code rate. They use generalized dinormalization (GDN) as the network structure, which includes a linear mapping followed by a nonlinear parameter normalization. The effectiveness of GDN for images has been verified in existing designs. Ballé et al. then proposed an improved version, in which they used 3 convolutional layers, each followed by a downsampling layer and a GDN layer as a forward transform. Therefore, they used 3 layers of inverse GDN, each followed by an upsampling layer and a convolutional layer to motivate the inverse transform. Additionally, an arithmetic coding method is designed to compress integer codes. It is reported that its performance is better than JPEG and JPEG 2000 on the Kodak dataset. In addition, Ballé et al. improved the method by designing a scale hyper-prior in the automatic codec. They used a subnet h a Transform the latent representation y into z = h a (y), and z is quantized and sent as side information. Accordingly, the inverse transform is performed through the subnet h s Implementation, try to extract the quantized side information Decode the quantized The standard deviation of further used in the arithmetic coding process of . On the Kodak image set, their method is slightly worse than BPG in terms of PSNR. D.Minnen et al. further exploited the structure of the residual space by introducing an autoregressive model to estimate the standard deviation and mean. In the existing design, Z.Cheng et al. used a Gaussian mixture model to further remove redundancy in the residual. It is reported that on the Kodak image set, when PSNR is used as the evaluation metric, its performance is comparable to VVC. 2.3.3. Hyper-prior model In the transform codec approach to image compression, the encoder subnetwork (Section 2.3.2) uses a parametric analysis transform The image vector x is transformed into a latent representation y, which is then quantized as In the form of is a discrete value that can be losslessly compressed and transmitted as a bit sequence using entropy coding techniques, such as arithmetic coding. As from Figure 3 The middle left image and the middle right image can clearly see that the elements There is a significant spatial dependence between them. It is worth noting that their scales (middle right image) appear to be coupled in the spatial domain. In the existing design, another set of random variables is introduced to capture spatial dependencies and further reduce redundancy. In this case, the image compression network Figure 4 Shown in. exist Figure 4 The left hand side of the model is the encoder g a and decoder g s (Explained in Section 2.3.2). The right hand side is used to obtain In this architecture, the encoder and decoder take the input image x and feed it into g a , thus generating a response y with a spatially varying standard deviation. The response y is sent to h a , to summarize the standard deviation distribution of z. Then z is quantized as is compressed and transmitted as side information. The encoder then uses the quantized vector To estimate the spatial distribution of the standard deviation σ, and use it to compress and transmit the quantized image representation The decoder first recovers the compressed signal Then it uses h S to obtain σ, which provides the correct probability estimate for successful recovery Then Feed g s to obtain a reconstructed image. When the super encoder and super decoder are added to the image compression network, the quantized latent The airspace redundancy is reduced. Figure 3 The rightmost image in corresponds to the quantized latent when using a super encoder / decoder. Compared to the middle right image, the spatial redundancy is significantly reduced because the samples of the quantized latent are less correlated. Figure 3 An image from the Kodak dataset and different representations of the image are shown. Figure 3 The leftmost image in the figure shows an image from the Kodak dataset. Figure 3 The middle left image in shows a visualization of the latent representation y of the image. Figure 3 The middle right image in shows the standard deviation σ of the latent quantity. Figure 3 The rightmost image in shows the latent y after introducing the super (super encoder and decoder) network. Figure 4 The network architecture of the autoencoder implementing the super-prior model is shown. The image autoencoder network is shown on the left, and the right corresponds to the super-prior subnetwork. The analysis and synthesis transformations are denoted as g a and g s Q represents quantization, AE and AD represent arithmetic encoder and arithmetic decoder respectively. The super prior model consists of two sub-networks, the super encoder (with h a denoted) and super decoder (with h s The super prior model generates quantized super implicit It includes the implicit quantity Information about the probability distribution of samples. is included in the bitstream and Sent together to the receiving end (decoder). 2.3.4. Context Model Although the hyper-prior model improves the quantized implicit , but additional improvement can be obtained by exploiting autoregressive models that predict quantized latent quantities from their causal context (context model). The term autoregressive means that the output of a process is later used as its input. For example, the context model subnetwork generates one sample of the latent quantity, which is later used as input to get the next sample. Existing designs utilize a joint architecture where both the super-prior model subnetwork (super-encoder and super-decoder) and the context model subnetwork are utilized. The super-prior and context models are combined to learn a quantized latent The probability model of is then used for entropy coding and decoding. Figure 5As shown, the output of the context sub-network and the super decoder sub-network is a sub-network combination called the entropy parameter, which generates the mean μ and scale σ (or variance) parameters of the Gaussian probability model. The Gaussian probability model is then used to encode the quantized latent samples into the bitstream with the help of the arithmetic encoder-decoder (AE) module. In the decoder, the Gaussian probability model is used to obtain the quantized latent from the bitstream through the arithmetic decoder (AD) module. Figure 5 The block diagram of the combined model is shown. The combined model jointly optimizes the autoregressive component, which estimates the probability distribution of the delay from its causal context (context model) together with the hyperprior and the underlying autoencoder. The real-valued latent representation is quantized (Q) to create a quantized latent and quantified super-implicit It is compressed into a bitstream using an arithmetic encoder (AE) and decompressed by an arithmetic decoder (AD). The highlighted areas correspond to the components executed by the receiving end (ie, the decoder) to recover the image from the compressed bitstream. Typically, latent samples are modeled as Gaussian distributions or Gaussian mixture models (not limited to). In existing designs, according to the accompanying drawings, context models and hyper-prior are used jointly to estimate the probability distribution of latent samples. Since the Gaussian distribution can be defined by mean and variance (also called σ or scale), the joint model is used to estimate the mean and variance (expressed as μ and σ). 2.3.5. Encoding and decoding process using joint autoregressive hyper-prior model Figure 5 Corresponding to the existing technical compression method. In this section and the next section, the encoding and decoding processes will be described respectively. Figure 6 The encoding process is depicted. The input image is first processed using the encoder subnetwork. The encoder transforms the input image into a transformed representation called the latent, denoted by y. Then y is input to the quantizer block denoted by Q to obtain the quantized latent after is converted into a bitstream (bits1) using an arithmetic coding block (denoted as AE). The arithmetic coding block converts the Each sample of is converted into a bit stream (bits1). The module super encoder, context, super decoder and entropy parameter sub-network are used to estimate the probability distribution of quantized latent samples. The hidden quantity y is input to the super encoder, and the super encoder outputs the super hidden quantity (denoted by z). The super hidden quantity is then quantized as The arithmetic encoding and decoding (AE) module is used to generate a second bit stream (bits2). The decomposition entropy module generates a probability distribution, which is used to encode the quantized super-hidden quantity into the bit stream. The quantized super-hidden quantity includes information about the quantized hidden quantity information about the probability distribution of . The entropy parameter subnetwork generates the quantized latent The information generated by the entropy parameters usually includes the mean μ and scale (or variance) σ parameters that are used together to obtain a Gaussian probability distribution. The Gaussian distribution of a random variable x is defined as Where parameter μ is the mean or expected value of the distribution (also its median and mode), and parameter σ is its standard deviation (or variance, or scale). To define a Gaussian distribution, the mean and variance need to be determined. In existing designs, the entropy parameter module is used to estimate the mean and variance values. The subnetwork superdecoder generates part of the information used by the entropy parameter subnetwork, and the other part of the information is generated by an autoregressive module called the context module. The context module generates information about the probability distribution of samples of the quantized latent quantity using samples that have been encoded by the arithmetic coding (AE) module. This is usually a matrix consisting of many samples. or The index to indicate the sample depends on the matrix Dimensions of . Sample Encoded by the AE one by one, usually using raster scan order. In raster scan order, the rows of the matrix are processed from top to bottom, where samples in a row are processed from left to right. In this case (where the AE encodes the samples into the bitstream using raster scan order), the context module generates the same samples as the samples using samples encoded previously in raster scan order. The information generated by the context module and the super decoder is combined by the entropy parameter module to generate the latent quantity Probability distribution encoded into the bitstream (bits1). Finally, as a result of the encoding process, the first bitstream and the second bitstream are sent to a decoder. Note that other names can be used for the above modules. In the above description, Figure 6 All the elements in are collectively called encoders. The analytical transformation that transforms the input image into a latent representation is also called an encoder (or autoencoder). 2.3.6. Decoding process using the joint autoregressive hyper-prior model Figure 7The decoding process is depicted separately. In the decoding process, the decoder first receives the first bit stream (bits1) and the second bit stream (bits2) generated by the corresponding encoder. Bits2 is first decoded by the arithmetic decoding (AD) module by using the probability distribution generated by the decomposed entropy subnetwork. The decomposed entropy module usually generates the probability distribution using a predetermined template, such as a predetermined mean and variance value in the case of a Gaussian distribution. The output of the arithmetic decoding process of bits2 is the quantized super-implicit The AD process is the inverse of the AE process applied in the encoder. The AE and AD processes are lossless, which means that the quantized super-hidden data generated by the encoder is Can be reconstructed at the decoder without any changes. In obtaining After that, it is processed by the super decoder, whose output is fed to the entropy parameter module. The three subnetworks, context, super decoder, and entropy parameters used in the decoder are the same as those in the encoder. Therefore, the exact same probability distribution can be obtained in the decoder as in the encoder, which is essential for reconstructing the quantized latent information without lossless As a result, the quantized implicit quantity obtained in the encoder can be obtained in the decoder The same version of . After the entropy parameter subnetwork obtains the probability distribution (e.g., mean and variance parameters), the arithmetic decoding module decodes the quantized latent samples one by one from the bitstream bits1. From a practical point of view, the autoregressive model (context model) is serial in nature and therefore cannot be accelerated using techniques such as parallelization. Finally, the fully reconstructed quantized implicit Input to the composite transform (in Figure 7 The decoder module is used to obtain the reconstructed image. In the above description, Figure 7 All elements in are collectively referred to as decoders. The synthetic transform that converts the quantized latent quantity into the reconstructed image is also called a decoder (or auto decoder). 2.4. Neural Networks for Video Compression Similar to traditional video codec technology, neural image compression is used as the basis for intra-frame compression in neural network-based video compression, so the development of neural network-based video compression technology is later than that of neural network-based image compression, but more efforts are needed to solve the challenges caused by its complexity. Since 2017, some researchers have been working on neural network-based video compression schemes. Compared with image compression, video compression requires effective methods to remove redundancy between pictures. Inter-picture prediction is then a key step in these works. Motion estimation and compensation are widely adopted, but have not been implemented by trained neural networks until recently. Research on neural network-based video compression can be divided into two categories according to the target scenario: random access and low latency. In the case of random access, decoding is required to start from any point in the sequence, usually dividing the entire sequence into multiple independent segments, and each segment can be decoded independently. In the case of low latency, it aims to reduce the decoding time, so that subsequent frames are usually decoded using only the previous frame in time as a reference frame. 2.4.1. Low latency Chen et al. first proposed a video compression scheme with a trained neural network. They first divided the video sequence frames into blocks, and each block will choose one of the two available modes, intra-frame codec or inter-frame codec. If intra-frame codec is selected, there is an associated autoencoder for compressing the block. If inter-frame codec is selected, motion estimation and compensation are performed using traditional methods, and the trained neural network will be used for residual compression. The output of the autoencoder is directly quantized and encoded by the Huffman method. Chen et al. proposed another neural network-based video codec scheme with PixelMotionCNN. Frames are compressed in temporal order and each frame is divided into blocks that are compressed in raster scan order. Each frame will first be extrapolated with the previous two reconstructed frames. When a block is to be compressed, the extrapolated frame is sent to PixelMotionCNN along with the context of the current block to derive the latent representation. The residual is then compressed by a variable bitrate image scheme. The performance of this scheme is comparable to H.264. Lu et al. proposed a real-time end-to-end neural network-based video compression framework in which all modules are implemented with neural networks. The scheme accepts the current frame and the previous reconstructed frame as input, and the optical flow will be derived as motion information using a pre-trained neural network. The motion information will be warped with the reference frame, followed by a neural network that generates motion compensated frames. Two separate neural autoencoders are used to compress the residual and motion information. The entire framework is trained with a single rate-distortion loss function. It achieves better performance than H.264. Rippel et al. proposed a video compression scheme based on advanced neural networks. It inherits and extends the traditional video codec scheme of neural networks with the following main features: 1) only one autoencoder is used to compress motion information and residuals; 2) multiple frames and multiple optical flows are used for motion compensation; 3) online states are learned and propagated through subsequent frames over time. This scheme achieves better performance than the HEVC reference software in MS-SSIM. J.Lin et al. proposed a video compression framework based on an extended end-to-end neural network. In this scheme, multiple frames are used as references. Therefore, it is possible to provide a more accurate prediction of the current frame by using multiple reference frames and associated motion information. Additionally, motion field prediction is used to remove motion redundancy in the temporal channel. A post-processing network is also introduced in this work to remove reconstruction artifacts from the previous process. The scheme outperforms H.265 in terms of both PSNR and MS-SSIM. Eirikur et al. proposed a scale-spatial flow to replace the commonly used optical flow by adding a scale parameter. It is reported that better performance than H.264 is achieved. Z.Hu et al. proposed a multi-resolution representation for optical flow. Specifically, the motion estimation network generates multiple optical flows with different resolutions and lets the network learn which one to choose under the loss function. The performance is slightly improved and better than H.265. 2.4.2. Random Access Wu et al. proposed a neural network-based video compression scheme with frame interpolation. Key frames are first compressed with a neural image compressor, and the remaining frames are compressed in a hierarchical order. They perform motion compensation in the perceptual domain, i.e., feature maps are derived at multiple spatial scales of the original frame, and the motion is used to warp the feature maps to be used for the image. The method is reported to have performance comparable to H.264. Djelouah et al. proposed a method for interpolation-based video compression, where the interpolation model combines motion information compression and image synthesis, and the same autoencoder is used for both the image and the residual. Amirhossein et al. proposed a neural network-based video compression method based on variational autoencoders and deterministic encoders. Specifically, the model consists of an autoencoder and an autoregressive prior. Unlike previous methods, this method accepts group of pictures (GOP) as input and encodes latent representations by considering temporal correlations, thereby incorporating a 3D autoregressive prior. The method provides performance comparable to H.265. 2.5. Preliminary knowledge Almost all natural images / videos are in digital format. A grayscale digital image can be represented as in is a set of pixel values, m is the image height, and n is the image width. For example, a common setting is in this case, Therefore, pixels can be represented by 8-bit integers. Uncompressed grayscale digital images have 8 bits per pixel (bpp), and the number of bits after compression is definitely less. Color images are usually represented in multiple channels to record color information. For example, in the RGB color space, an image can be represented as Three independent channels store red, green, and blue information respectively. Similar to an 8-bit grayscale image, an uncompressed 8-bit RGB image has 24bpp. Digital images / videos can be represented in different color spaces. Neural network-based video compression schemes are mainly developed in the RGB color space, while traditional codecs usually use the YUV color space to represent video sequences. In the YUV color space, the image is decomposed into three channels, namely Y, Cb, and Cr, where Y is the brightness component and Cb / Cr are the chrominance components. Since the human visual system is less sensitive to the chrominance components, Cb and Cr are usually downsampled for pre-compression. A color video sequence consists of multiple color images, called frames, to record scenes at different time stamps. For example, in the RGB color space, a color video can be represented as X = {x0, x1, …, x t ,…,x T-1}, where T is the number of frames in the video sequence, If m=1080, n=1920, And the video has 50 frames per second (fps), then the data rate of the uncompressed video is 1920×1080×8×3×50=2,488,320,000, about 2.32 Gbps, which requires a lot of storage and thus must be compressed before transmission over the Internet. Typically, for natural images, lossless methods can achieve a compression ratio of about 1.5 to 3, which is clearly lower than required. Therefore, lossy compression is developed to achieve further compression ratios, but at the expense of distortion. Distortion can be measured by calculating the average squared difference between the original image and the reconstructed image, i.e., the mean square error (MSE). For grayscale images, MSE can be calculated using the following equation. Therefore, the quality of the reconstructed image compared to the original image can be measured by the Peak Signal-to-Noise Ratio (PSNR): in yes The maximum possible value in , for 8-bit grayscale images, it is equal to 255. There are other quality assessment metrics, such as structural similarity (SSIM) and multi-scale structural similarity (MS-SSIM). To compare different lossless compression schemes, the compression ratio is compared at a given resulting rate or vice versa. However, to compare different lossy compression methods, both the rate and the reconstruction quality must be considered. For example, calculating the relative rate at multiple quality levels and then taking the average rate is a common approach; the average relative rate is called Bjontegaard's delta rate (BD-rate). There are other important aspects for evaluating image / video codec schemes, including encoding / decoding complexity, scalability, robustness, etc. 2.5.1. Quantization and Entropy Coding and Decoding The terms quantization and entropy coding can be defined as follows: In mathematics and digital signal processing, quantization is the process of mapping input values ​​from a large (usually continuous) set to output values ​​in a (countably) smaller set, usually with a finite number of elements. Rounding and truncation are classic examples of quantization processes. Almost all digital signal processing involves quantization to some extent, since the process of representing a signal in digital form usually involves rounding. Quantization also forms the core of essentially all lossy compression algorithms. The quantization equation can be as follows: in, is the quantized value, y is the sample to be quantized, and Δ is the quantization step size. For example, according to the above equation 1, if the quantization step size increases, the quantized value will become smaller and fewer bits will be required to encode the quantized samples. In other words, an increased quantization step size results in a coarser quantization. On the other hand, a smaller Δ (usually a positive number) results in a finer quantization and a higher number of bits to encode the quantized samples. Encoded into the bitstream. Entropy coding is a general lossless data compression method that encodes symbols using a number of bits that is inversely proportional to the symbol probability. This term is used to refer to some compression algorithms that are based on the probability distribution of the source symbols. Huffman coding and arithmetic coding are two typical representatives of this type of algorithm. In order to perform entropy coding, the statistical properties of the symbols (such as the probability distribution) must be understood. Figure 8 Two possible implementations of entropy coding, decoding and quantization in NN-based image compression are shown. Described from the encoder perspective. exist Figure 8In the figure, two possible implementations of an encoder for an NN-based image compression method are described. The encoder for the NN-based image compression method consists of four modules. The first module is an analysis transform. The analysis transform transforms the input image into an implicit representation y. The transformed representation y is usually easier to compress than the input image because the analysis transform is designed to eliminate redundancy in the input image signal. The second module is an estimation module, which estimates the statistical properties of the implicit representation. The statistical properties may include a mean (expressed as μ), a variance (expressed as σ), or higher-order moments. The third module is a quantization module, which converts a continuous implicit representation signal into a quantized discrete value. Finally, the fourth module is an entropy codec (expressed as EC), which converts the quantized symbol values ​​into a bit stream using the statistical properties generated by the estimation module. For the encoder, there are two slightly different implementation alternatives, depending on how the mean value μ is used. In the first implementation, the entropy coding module receives the mean value and other statistics (such as variance σ) as input and uses it to encode the quantized hidden symbols into the bitstream. In the second implementation alternative, the mean value is subtracted from the hidden symbols y to obtain residual samples w. The residual samples are then quantized by the quantization module to obtain quantized residual samples The entropy coding module converts the quantized residual samples into a bitstream using the statistical information generated by the estimation module. The statistical information may include, for example, the variance σ. Fig. 9 Shown for Figure 8 The corresponding decoder implementation for the encoder implementation shown in . Fig. 9 Two possible alternative implementations of a decoder in a NN-based video compression method are described. According to the first alternative, the estimation module estimates the mean μ and other statistical properties (e.g., variance σ). The variance and optionally other statistical information are used by an entropy codec (denoted as EC) to decode the bitstream into quantized residual samples. Then the mean μ and the quantized residual samples are estimated are added together to obtain the quantized latent samples Finally, it is transformed into a reconstructed picture through a synthetic transformation. The first alternative decoder is implemented in Fig. 9 is depicted in the upper flow chart. According to a second alternative, the estimation module estimates the mean μ and other statistical properties (e.g. variance σ). The mean, variance and optionally other statistical information are used by the entropy codec (denoted EC) to decode the bitstream into quantized hidden samples Finally, it is transformed into a reconstructed picture through a synthetic transformation. The second alternative decoder is implemented in Fig. 9 is depicted in the bottom flow chart. In the above, the quantized latent sample is given by It should be noted that a sample is not necessarily a scalar value, it may be a vector and may contain multiple elements. In the rest of the application, the sample can be represented by or In the latter, “:” is used to indicate the presence of a third dimension and to emphasize that a sample has multiple elements. 3. Question 3.1. Core Issues In NN-based image or video compression architectures, most modules are highly parallelized (e.g. Fig. 9 ), so these processes can be greatly accelerated by graphics processing units (GPUs). On the other hand, the entropy coding (or decoding) process involves converting the input samples into a series of bits (i.e., a bitstream), and is therefore essentially a sequential process. Due to its sequential nature, the entropy coding module cannot be parallelized, so it becomes a bottleneck process and is the reason why NN-based compression methods are slower. 4. Detailed solution The detailed solutions below should be considered as examples to explain the general concept. These solutions should not be interpreted in a narrow way. In addition, these solutions can be combined in any way. 4.1. Objectives of the project This solution aims to skip the entropy encoding and decoding process under certain conditions to reduce the time spent on entropy encoding and decoding, thereby increasing encoding / decoding speed. 4.2. Solution Core According to the core of the solution, samples of y and / or w are divided into at least two subsets based on their statistical properties. In the encoder, samples of the first subset are encoded into the bitstream, while samples of the second subset are not encoded into the bitstream. In the decoder, samples of the first subset are decoded from the bitstream, while samples of the second subset are not decoded from the bitstream, and their values ​​are inferred. 4.2.1. Encoding process According to the plan: The input image is transformed by the first sub-network to obtain quantized latent samples (or the quantized residual implicit sample). The statistical parameter σ (eg, variance parameter) is estimated by the estimation module. Quantified implicit quantity (or residual implicit ) are divided into at least two subsets according to the estimated statistical parameter σ, and (or and ), where each subset includes at least one sample. Belongs to the first subset The quantized implicit quantity of is encoded by using the entropy coding module of σ and included in the bitstream. 4.2.2. Decoding process According to the plan: The statistical parameter σ (eg, variance parameter) is estimated by the estimation module. Quantified implicit quantity (or residual implicit ) are divided into at least two subsets according to σ, where the first subset (or ) are decoded from the bitstream and the second subset (or ) will be inferred. Belongs to the first subset (or ) are decoded from the bitstream according to σ. Belongs to the second subset (or )’s values ​​are inferred. Quantified latent samples (or quantized residual latent samples ) is processed by a second sub-network (e.g., a synthetic transform) to obtain a reconstructed picture. 4.3. Solution details Fig.10 The implementation of the solution at the NN-based image encoder is depicted. First, the input image is transformed by an analysis transform to obtain samples of the latent representation y. Then, the statistical parameters are estimated by an estimation module. The parameters may include a mean parameter μ representing the mean of the probability distribution and / or a variance parameter σ representing the variance value of the probability distribution. In one implementation of the solution (e.g. Fig.10 As shown in the right figure), first subtract the mean μ from the sample of the latent quantity y to obtain the residual latent quantity sample w. Then use the quantization module (denoted as Q) to quantize the latent quantity or residual latent quantity sample to obtain the quantized sample or The grouping module uses a statistical parameter (e.g., σ) to divide the quantized samples into two groups, and And only the first subset (or ) are sent to the entropy codec (denoted as EC). (or ) are not sent to the entropy codec and are therefore not included in the bitstream. The samples sent to the entropy codec are processed by the entropy codec and are included in the bitstream. The samples not sent to the entropy codec are not processed by the entropy codec or are ignored by it. Therefore, they are not included in the bitstream. Fig.10 Two possible implementations of the encoder are shown, and Fig.11 Two possible implementations of the decoder are shown. Fig.11 The implementation of the solution at the NN-based image decoder is depicted. First, statistical parameters are obtained by an estimation module. The parameters may include a mean parameter μ representing the mean value of the probability distribution and / or a variance parameter σ representing the variance value of the probability distribution. For example, the estimation module may include a neural network, and the input of the estimation module may be a bitstream. Using the statistical parameters obtained by the estimation module, the entropy encoding and decoding module decodes the first subset (e.g., ) is the quantized residual implicit (or quantified implicit ). The grouping module is configured to determine the samples corresponding to Which samples of the first subset and the remaining samples (belonging to the second subset) are inferred. The grouping module uses statistical parameters (eg, μ and / or σ) to determine which sample corresponds to which subset. The inference about the value of the samples belonging to the second subset may include one of the following: 1. Set the values ​​of samples belonging to the second subset equal to zero. 2. Set the values ​​of the samples belonging to the second subset equal to the corresponding mean values ​​μ estimated by the estimation module. 3. Set the values ​​of samples belonging to the second subset equal to a scalar constant. For example, in Fig.11 In the bottom figure, the entropy coding (EC) module receives the bitstream and decodes The entropy encoding and decoding module decodes the value using the statistical parameters obtained by the estimation module. Include only A subset of samples, so the grouping module is used to determine Which samples are included in the subset and which samples are not decoded by the entropy encoding and decoding process. The grouping module uses statistical parameters (such as variance σ) to determine which samples are decoded by EC and which samples are not decoded. There may be corresponding (or ) for each sample. The determination of whether a sample is EC decoded can be based on a threshold operation, where the statistical parameter value σ[i,j] or its function f(σ[i,j]) is compared with the threshold. If the result of the comparison is true, the position (or ) is set equal to Otherwise, the position (or ) is inferred. In one example, the value of a sample can be inferred to be equal to μ[i,j], which is the value corresponding to position (or ) is the average value of the samples at ). In another example, the value of the sample can be inferred to be equal to a predetermined constant value, such as zero. After the values ​​of the samples in the first subset are decoded from the bitstream, and the samples in the second subset are inferred, is reconstructed. Then The transformed image is processed to obtain a reconstructed image. According to this solution, the grouping process of samples (grouped into subsets) is based on estimated statistical parameters. This is particularly useful because the statistical parameters indicate how "unclear" the values ​​of the samples are. For example, Fig.12 The probability distribution of a sample is illustrated when the variance parameter is small (on the left) and when the variance parameter is large (on the right). On the left, the sample is likely to have a value equal to or very close to zero. And, since the likelihood at sample values ​​of 3 or -3 is close to 0, the likelihood of these values ​​occurring is very low. Informally, we can say that the sample is "clearer" because it is likely to be 0. On the other hand, when the variance parameter is large ( Fig.12 When the sample values ​​are more diverse (as shown in the right figure in Figure 2), the sample values ​​can be more diverse. On the right side, the probability that the sample value is not equal to 0 is quite large. Therefore, the sample is "unclear". Fig.12 Two possible probability distributions are shown: (on the left) the probability distribution of the sample when the variance of the probability distribution is small, and (on the right) the probability distribution of the sample when the variance of the probability distribution is large. This solution exploits the above observations by grouping samples according to the statistical parameters they correspond to. For example, if a sample is likely to be zero (small variance), it is grouped into one subset, while if the variance is large, it is grouped into another subset. When the variance is small, the value of the signal is "clear" and is likely to be zero (or some other mean value that is in the center of the probability distribution). In this case, skipping the entropy encoding and decoding process for this sample will not significantly change the result, since the value is already almost certain. On the other hand, if the variance is large, the value of the sample may vary (its value is "unclear"), so in this case it is valuable to encode these samples into the bitstream. The estimation module may include a sub-network based on a neural network. For example, the input of the estimation module may be a bit stream, and the output of the sub-network may be a mean and variance parameter of a probability distribution. More specifically, the output of the estimation module may be a mean and variance parameter of a Gaussian probability distribution. Example implementation of the solution: 1. Dividing the samples of the quantized latent quantity (or the quantized residual latent quantity) into at least two subsets includes: - grouping the quantized latent samples into blocks of size N×M, - checking whether the average of the variances corresponding to all samples included in the block is less than a threshold (or greater than a threshold), - If the result of the check is true, the samples of the block are grouped into a first subset, and if the result is false, the samples of the block are grouped into a second subset. 2. Dividing the samples of the quantized latent quantity (or the quantized residual latent quantity) into at least two subsets including: - grouping the quantized latent samples into blocks of size N×M, - checking whether the variance corresponding to all samples included in the block is smaller than a threshold (or larger than a threshold), - If the result of the check is true, the samples of the block are grouped into a first subset, and if the result is false, the samples of the block are grouped into a second subset. 3. Dividing the samples of the quantized latent quantity (or the quantized residual latent quantity) into at least two subsets includes: - using the variance parameter to obtain the probability corresponding to each sample of the quantized latent quantity, - grouping the quantized latent samples into blocks of size N×M, - check if the probability of all samples being included in the block is less than a threshold (or greater than a threshold), - If the result of the check is true, the samples of the block are grouped into a first subset, and if the result is false, the samples of the block are grouped into a second subset. 4. Dividing the samples of the quantized latent quantity (or the quantized residual latent quantity) into at least two subsets includes: - using the variance parameter to obtain the probability corresponding to each sample of the quantized latent quantity, - grouping the quantized latent samples into blocks of size N×M, - check if the average probability of all samples included in the block is less than a threshold (or greater than a threshold), - If the result of the check is true, the samples of the block are grouped into a first subset, and if the result is false, the samples of the block are grouped into a second subset. In Examples 3 and 4, the probability of a sample is compared to a threshold to determine whether the sample belongs to the first set or the second set. In one example, the probability of a sample is calculated as the probability that the value of the sample is equal to 0. In another example, the probability of a sample is calculated as the probability that the value of the sample is equal to the average value estimated by the estimation module. In another example, the probability of a sample is calculated as the probability that the value of the sample is equal to the average value estimated by the estimation module. The probability is the probability that the value of this sample is equal to the mean μ[i,j] estimated by the estimation module. In this example, the estimation module generates Each sample corresponds to the mean value μ[i,j]. The probability of a sample can be obtained using statistical parameters generated by the estimation module. For example, the estimation module can generate mean parameters and variance parameters for a Gaussian distribution, which is a probability distribution that can be used to estimate the probability that a sample is equal to any value. 4.4. Benefits of the solution The scheme skips the entropy encoding and decoding process according to statistical parameters to reduce the time spent on entropy encoding and decoding, thereby improving the encoding / decoding speed. 5. Examples 1. Decoder embodiment: A method for decoding an image or video, comprising at least one of the following steps: - obtaining at least one statistical parameter σ using an estimation module. - Use the bitstream and σ to get at least one sample value, expressed as - Determine the quantified hidden quantity (or ) is decoded from the bitstream or inferred based on σ. - If the sample is determined to be decoded, set the value of the sample equal to Otherwise, infer the value of the sample. Using quantized implicit (or ) to obtain a reconstructed image. 2. Encoder embodiment: A method for encoding an image or video, comprising the steps of: -First, transform the input image using the analysis transform to obtain the quantized latent (or ) samples. - Use the estimation module to obtain the statistical parameters σ. - Determine the quantized implicit quantity according to the statistical parameter σ (or ) are encoded into the bitstream. If the quantified implicit quantity is determined (or ) will be encoded, and the value is included in the bitstream according to the statistical parameter σ by the entropy codec. If it is determined that the value is not to be encoded, the value of the sample is not included in the bitstream. 3. According to embodiments 1 and 2, The estimation module includes a neural sub-network. 4. According to all the above embodiments, The estimation module receives a bitstream as input, processes the input using a neural subnetwork and outputs statistical parameters. 5. According to embodiment 1, Determine the quantified implicit quantity (or ) is to be decoded (or encoded) includes comparing at least one statistical parameter (or a function thereof), which at least one statistical parameter (or a function thereof) may be associated with at least one threshold. 6. According to all the above embodiments, The statistical parameter is the variance parameter. 7. According to embodiments 5 and 6, The value of the threshold (or an indication thereof) is included in the bitstream. 8. According to all the above embodiments, The variance parameter is the variance of the probability distribution (eg, a Gaussian probability distribution or a Laplace probability distribution). 9. According to all the above embodiments, Determine the quantified implicit quantity (or ) is to be decoded (or encoded) including at least one of the following steps: - use statistical parameters to obtain probabilities, and -Compare the probability to a threshold. 10. According to all the above embodiments, Determine the quantified implicit quantity (or ) is to be decoded (or encoded) including at least one of the following steps: - using the variance parameter to obtain the probability corresponding to a sample of the quantized latent quantity, - grouping the quantized latent samples into blocks of size N×M, - check if the probability of all samples being included in the block is less than a threshold, If the result of the check is true, it is determined that the samples included in the block are to be decoded (or encoded). 11. According to all the above embodiments, Determine the quantified implicit quantity (or ) is to be decoded (or encoded) including at least one of the following steps: - using the variance parameter to obtain the probability corresponding to a sample of the quantized latent quantity, - grouping the quantized latent samples into blocks of size N×M, - check if the average of the probabilities of all samples included in the block is less than a threshold, If the result of the check is true, it is determined that the samples included in the block are to be decoded (or encoded). 12. According to all embodiments 11, The obtained probability is the probability that the quantized sample is equal to zero. 13. According to all the above embodiments, Determine the quantified implicit quantity (or ) is to be decoded (or encoded) including at least one of the following steps: - grouping the quantized latent samples into blocks of size N×M, - checking whether the variance values ​​corresponding to all samples included in the block are smaller than a threshold, If the result of the check is true, it is determined that the samples included in the block are to be decoded (or encoded). 14. According to all the above embodiments, Determine the quantified implicit quantity (or ) is to be decoded (or encoded) including at least one of the following steps: - grouping the quantized latent samples into blocks of size N×M, - checking whether the average of the variances corresponding to all samples included in the block is less than a threshold, If the result of the check is true, it is determined that the samples included in the block are to be decoded (or encoded). 15. According to all the above embodiments, If it is determined that a sample is not decoded from the bitstream, its value is inferred to be equal to a predetermined value. 16. According to all the above embodiments, If it is determined that a sample is not decoded from the bitstream, its value is inferred to be equal to the mean value. 17. According to embodiment 16, The average value is obtained by the estimation module. 18. According to all the above embodiments, The probability of a sample is the probability that the value of the sample is equal to zero. 19. According to all the above embodiments, The probability of a sample is the probability that the value of the sample is equal to the mean estimated by the estimation module. 20. According to embodiments 10 to 14, Where M and N can be positive integers. 21. According to embodiments 10 to 14 and 19, where M and N may be signaled in the bit stream. 22. According to any of the above embodiments, the value of at least one segment of control information is included, and the control information can determine whether and / or how to apply any of the above methods. 23. Whether and / or how to apply any of the above methods may depend on conditions, such as color format and / or color components. 24. According to any of the above embodiments, any value included in the bitstream can be encoded and decoded at the sequence level / picture level / slice level / block level. 25. According to any of the above embodiments, any value included in the bitstream may be binarized before being encoded or decoded. 26. According to any of the above embodiments, any value included in the bitstream can be encoded using at least one arithmetic coding context.

[0040] More details of embodiments of the present disclosure related to neural network-based visual data encoding and decoding are described below.As used herein, the term "visual data" may refer to an image, a picture in a video, or any other visual data suitable for encoding and decoding.

[0041] As discussed above, in existing architectures, most modules can be massively parallelized, so these processes can be significantly accelerated by GPUs. On the other hand, the process of entropy coding and decoding (encoding or decoding) consists of the conversion between input samples and bit sequences (i.e., bitstreams), and is therefore inherently a sequential process. Due to its sequential nature, entropy coding and decoding is not parallelizable, so it is a bottleneck process and slows down the entire coding and decoding process.

[0042] In order to solve the above problems and some unmentioned problems, a visual data processing solution as described below is disclosed.

[0043] Fig.13 An example visual data encoding and decoding process 1300 according to some embodiments of the present disclosure is shown. For example, the visual data encoding process 1300 may be performed by Figure 1 The visual data encoder 114 shown executes. It should be understood that the visual data encoding process 1300 may also include additional blocks not shown, and / or may omit the blocks shown. The scope of the present disclosure is not limited in this regard.

[0044] like Fig.13 As shown, at the first neural network 1312, an analytical transformation can be performed on the visual data 1310 to obtain an implicit representation of the visual data 1310 (in Fig.13 y in the figure). Visual data 1310 may include one or more pictures in an image or video. At block 1314, the statistic may be subtracted from the latent representation to obtain a residual latent representation (in Fig.13 In the expression ). The statistical value may be generated by the second neural network 1322 and indicates the prediction of the latent representation. As an example and not limitation, the statistical value may be the average value of a probability distribution, such as a Gaussian probability distribution, (in Fig.13 denoted as μ in the figure). For example, a probability distribution may describe the probability distribution of the values ​​of one or more samples of the implicit representation. It should be understood that a statistical value may also be referred to as a statistical parameter, a probability parameter, a probability distribution parameter, etc. The scope of the present disclosure is not limited in this respect.

[0045] The statistical value may be included in the statistical information generated by the second neural network 1322. For example, the second neural network 1322 may be referred to as an estimation model. It should be noted that the model may also be referred to as a module in the present application. That is, the second neural network may also be referred to as a first module, and the estimation model may also be referred to as an estimation module. The input of the second neural network 1322 may include an implicit representation, and the output of the second neural network 1322 includes statistical information. In one example, the second neural network 1322 may include a neural network-based subnetwork. In another example, the second neural network 1322 may include a first subnetwork for generating a statistical value and a second subnetwork for generating another statistical value. As an example and not limitation, the other statistical value may be the variance (in Fig.13 denoted as σ in ). For example, the probability distribution may describe the probability distribution of the values ​​of one or more samples of the implicit representation. The first subnetwork may be a superdecoder subnetwork, and the second subnetwork may be a superscale decoder subnetwork. In one example, the second neural network 1322 may also include a superencoder subnetwork. It should be understood that the above-mentioned mean and variance are merely two specific examples of statistical values ​​that may be used. Any other suitable statistical value may also be used, such as a standard deviation. The scope of the present disclosure is not limited in this regard.

[0046] At quantization block 1316, the residual implicit representation may be quantized to obtain a quantized residual implicit representation. The quantized residual implicit representation may also be referred to as a quantization of the residual implicit representation. In one example, the residual implicit representation may be quantized by using a rounding function. In another example, the residual implicit representation may be quantized by using a floor function. Alternatively, the residual implicit representation may be quantized by using a ceiling function. It should be understood that the above examples are described for descriptive purposes only. The scope of the present disclosure is not limited in this regard.

[0047] At grouping block 1318, at least one sample may be determined from the quantization of the representation based on statistical information. The at least one sample constitutes a group, the values ​​of which are not encoded into the bitstream. Therefore, the process for determining at least one sample may also be referred to as a grouping process, during which at least one group of samples may be acquired. In one example, the grouping process may be performed at the sample level. In other words, the determination of at least one sample is performed for each sample of the quantized residual implicit representation. Alternatively, the grouping process may be performed at the block level. More specifically, the samples of the quantized residual implicit representation may be divided into a plurality of blocks. Each of the plurality of blocks may have a predetermined size of N times M, such as 8×8. Each of N and M is a positive integer. N and M may be indicated in the bitstream. Samples may be grouped based on a plurality of blocks. In other words, the determination of at least one sample is performed for each of the plurality of blocks, and samples in the same block are divided into the same set.

[0048] In some embodiments, for the quantized samples, if the first statistic corresponding to the sample is less than the first threshold, it can be determined that the at least one sample includes the sample, and the sample can be grouped into the second set. Otherwise, it can be determined that the sample will not be included in the at least one sample, and the sample can be grouped into a first set different from the second set. In one example, the first statistic can be the variance described above.

[0049] Alternatively, for the quantized sample, if the first statistical value corresponding to the sample is greater than another threshold value that may be the same as or different from the first threshold value, it can be determined that the at least one sample includes the sample, and the sample can be grouped into the second set. Otherwise, it can be determined that the sample will not be included in the at least one sample, and the sample can be grouped into the first set. The above comparison can be performed for each sample of the quantized residual implicit representation to obtain the at least one sample.

[0050] It should be understood that at least one sample can be determined in any other suitable manner, such as based on a comparison between a threshold value and a value determined based on a first statistical value, or a comparison between a threshold value and a probability that the value of the sample is equal to a target value. The probability can be determined based on statistical information. The target value of the sample can be a first predetermined value (such as zero) or a second statistical value (such as an average value) corresponding to the sample. This is described in detail at the above section 4.3.

[0051] In some alternative embodiments, the grouping process may be performed at the block level. In this case, for samples in the quantized block, if the first statistic corresponding to each sample in the block is less than a fourth threshold, it may be determined that at least one sample includes all samples in the block, and these samples may be grouped into the second set. Otherwise, it may be determined that these samples will not be included in at least one sample, and these samples may be grouped into a first set different from the second set. In one example, the first statistic may be the variance described above.

[0052] Alternatively, for samples in the quantized block, if the first statistical value corresponding to each sample in the block is greater than another threshold value that may be the same as or different from the fourth threshold value, it may be determined that at least one sample includes all samples in the block, and these samples may be grouped into the second set. Otherwise, it may be determined that these samples will not be included in at least one sample, and these samples may be grouped into the first set. The above comparison may be performed for each block of the quantized residual implicit representation to obtain at least one sample.

[0053] It should be understood that at least one sample can be determined in any other suitable manner, such as based on a comparison between a threshold and a first metric determined based on a first statistical value corresponding to the sample in the block, or a comparison between a threshold and the probability that the value of each sample in the block is equal to a target value, or a comparison between a threshold and a second metric determined based on the probability that the value of the sample in the block is equal to the corresponding target value. As an example and not limitation, the first metric can be an average of the first statistical values, a minimum first statistical value of the first statistical values, or a maximum first statistical value of the first statistical values. The second metric can be an average of the probabilities, a minimum probability of the probabilities, or a maximum probability of the probabilities. At least one threshold can be indicated in the bitstream.

[0054] In some embodiments, the output of the grouping block 1318 may include the remaining samples of the quantized residual implicit representation in addition to the at least one sample. For example, the output of the grouping block 1318 may include the above-mentioned first group of samples (in Fig.13 In the expression ). At entropy encoder 1320, an entropy encoding process may be performed on the first set of samples based on the statistical information generated by the second neural network. Fig.13In the example shown, only variance is used for the entropy encoding process, any other suitable statistical value may be used, such as mean and / or standard deviation. The entropy encoding process performed by the entropy encoder 1320 may be an arithmetic encoding process, a Huffman encoding process, or the like.

[0055] Additionally, the statistical information generated by the second neural network can be encoded into a bit stream by performing entropy coding and decoding on the statistical information. Fig.13 Not shown in FIG.

[0056] In a special case, the at least one sample may include all samples implicitly represented by the quantized residual. In this case, the values ​​of all samples implicitly represented by the quantized residual will not be encoded into the bitstream, but the statistical information is signaled in the bitstream.

[0057] In view of the above, the value of at least one sample of the quantization of the representation of the visual data is not included in the bitstream. Thus, at least part of the quantization can be skipped during the entropy coding and decoding process to reduce the time spent on entropy coding and decoding. In this way, the proposed method can advantageously improve the coding and decoding speed and coding and decoding efficiency.

[0058] Despite the above Fig.13 An example visual data encoding process is described, but it should be understood that any other suitable variations of the visual data encoding process are also conceivable according to the present disclosure. In another example, an implicit representation other than the residual implicit representation may be adjusted, which is illustratively Fig.10 is shown in the left subfigure of .

[0059] Fig.14 An example visual data decoding process 1400 according to some embodiments of the present disclosure is shown. For example, the visual data decoding process 1400 may be performed by Figure 1 The visual data decoder 124 shown executes. It should be understood that the visual data decoding process 1400 may also include additional blocks not shown, and / or may omit the blocks shown. The scope of the present disclosure is not limited in this regard.

[0060] like Fig.14As shown, statistical information may be generated by the second neural network 1422. The statistical information may include a mean μ and / or a variance σ. For example, the second neural network 1422 may be referred to as an estimation model. The input of the second neural network 1422 may include a bitstream, and the output of the second neural network 1422 includes at least one statistical value. In one example, the second neural network 1422 may include a subnetwork based on a neural network. In another example, the second neural network 1422 may include a first subnetwork for generating a first statistical value and a second subnetwork for generating a second statistical value. In one example, the first statistical value may be the variance of a Gaussian probability distribution, and the second statistical value may be the mean of a Gaussian probability distribution. The first subnetwork may be a superscale decoder subnetwork, and the second subnetwork may be a superdecoder subnetwork. It should be understood that the above-mentioned mean and variance are only two specific examples of statistical values ​​that can be used. Any other suitable statistical value may also be used, such as a standard deviation. The scope of the present disclosure is not limited in this regard.

[0061] At the entropy decoder 1418, an entropy decoding process may be performed on the bitstream based on the statistical information to obtain a quantized residual implicit representation (in Fig.14 In the expression ) is the value of the first set of samples.

[0062] At grouping block 1416, at least one sample may be used with respect to Fig.13 The grouping process described in the grouping block 1318 in FIG. 1 is determined in a similar manner based on statistical information from the quantization of the residual latent representation, and therefore for the sake of brevity this will not be described in detail.

[0063] Since the value of at least one sample is not signaled in the bitstream, the value of each sample in the at least one sample can be inferred. In one example, the value of each sample in the at least one sample can be determined as a predetermined value. The predetermined value can be a scalar constant, such as zero. Alternatively, the value of each sample in the at least one sample can be determined as a third statistical value (e.g., an average value) corresponding to the sample. It should be understood that the value of each sample in the at least one sample can be inferred in any other suitable manner. The scope of the present disclosure is not limited in this regard. The quantized residual implicit representation can be reconstructed at the grouping block 1416 based on the acquired values ​​of the first group of samples and the inferred value of at least one sample. In the above special case, the quantized residual implicit representation can be reconstructed at the grouping block 1416 based on the inferred value of at least one sample.

[0064] At block 1414, the quantized implicit representation (in Fig.14 In the expression ) can be obtained by converting the statistical value (for example, Fig.14The average value μ in y is added to the output of the grouping block 1416 (ie, the quantized residual latent representation). At the synthesis transformation subnetwork 1412, a synthesis transformation can be performed on the quantized latent representation to obtain the reconstructed visual data 1410.

[0065] In view of the above, the value of at least one sample of the quantization of the representation of the visual data is not included in the bitstream. Thus, the proposed method enables at least a portion of the quantization to be skipped during the entropy coding and decoding process so as to reduce the time spent for the entropy coding and decoding. In this way, the proposed method can advantageously improve the coding and decoding speed and coding and decoding efficiency.

[0066] Despite the above Fig.14 An example visual data decoding process is described, but it should be understood that any other suitable variations of the visual data decoding process are also conceivable according to the present disclosure. In another example, an implicit representation instead of a residual implicit representation may be used, which is illustratively shown in Fig.11 is shown in the bottom subfigure of .

[0067] It should be understood that the above illustrations and / or examples are described only for the purpose of description. The scope of the present disclosure is not limited in this regard. The embodiments of the present disclosure should be considered as examples for explaining general concepts and should not be interpreted in a narrow manner. In addition, these embodiments can be applied individually or in combination in any way.

[0068] Fig.15 1 shows a flow chart of a method 1500 for visual data processing according to some embodiments of the present disclosure. The method 1500 may be implemented during conversion between visual data and a bit stream of visual data. Fig.15 As shown, method 1500 begins at 1502, where statistical information associated with a first representation of visual data is obtained. The first representation is generated based on applying a first neural network to the visual data. In one example, the first neural network can be used to perform an analytical transformation on the visual data. In some embodiments, the first representation can be an implicit representation of the visual data or a residual implicit representation of the visual data.

[0069] At 1504, at least one sample is determined from a second representation of the visual data based on the statistical information. A value of the at least one sample is not in the bitstream. The second representation is obtained by quantizing the first representation.

[0070] At 1506, a conversion is performed based on the determination. In one example, the conversion may include encoding the visual data into a bitstream. Alternatively or additionally, the conversion may include decoding the visual data from the bitstream. It should be understood that the above description is described for illustrative purposes only. The scope of the present disclosure is not limited in this regard.

[0071] In view of the foregoing, the value of at least one sample of the second representation of the first representation of the visual data is not included in the bitstream. Thus, the proposed method enables at least a portion of the second representation to be skipped during the entropy encoding and decoding process so as to reduce the time spent on entropy encoding and decoding. In this way, the proposed method can advantageously improve the encoding and decoding speed and encoding and decoding efficiency.

[0072] In some embodiments, at 1504, if it is determined that at least one of the following conditions is satisfied, it can be determined that at least one sample includes a sample of the second representation: a first statistical value corresponding to the sample is less than a first threshold, a value determined based on the first statistical value is less than a second threshold, or a probability that the value of the sample is equal to the target value is less than a third threshold.

[0073] In some embodiments, at 1504, if it is determined that at least one of the following conditions is satisfied, it can be determined that the at least one sample includes a sample in a block of the second representation: a first statistical value corresponding to each sample in the block is less than a fourth threshold, a first metric is less than a fifth threshold, the first metric is determined based on the first statistical value corresponding to the sample in the block, a probability that the value of each sample in the block is equal to a target value is less than a sixth threshold, or a second metric is less than a seventh threshold, and the second metric can be determined based on the probability that the value of the sample in the block is equal to the corresponding target value.

[0074] In some embodiments, at 1504, if it is determined that at least one of the following conditions is satisfied, it can be determined that at least one sample includes a sample of the second representation: a first statistical value corresponding to the sample is greater than an eighth threshold, a value determined based on the first statistical value is greater than a ninth threshold, or a probability that the value of the sample is equal to the target value is greater than a tenth threshold.

[0075] In some embodiments, at 1504, if it is determined that at least one of the following conditions is satisfied, it can be determined that the at least one sample includes a sample in a block of the second representation: a first statistical value corresponding to each sample in the block is greater than an eleventh threshold, the first metric is greater than a twelfth threshold, the first metric is determined based on the first statistical value corresponding to the sample in the block, the probability that the value of each sample in the block is equal to the target value is greater than a thirteenth threshold, or the second metric is greater than a fourteenth threshold, and the second metric can be determined based on the probability that the value of the sample in the block is equal to the corresponding target value.

[0076] In some embodiments, the first metric may be an average of the first statistical values, a minimum first statistical value among the first statistical values, or a maximum first statistical value among the first statistical values. Alternatively or additionally, the second metric may be an average of the probabilities, a minimum probability among the probabilities, or a maximum probability among the probabilities.

[0077] In some embodiments, the size of the block can be N times M, and each of N and M can be a positive integer. In one example, N and M can be indicated in the bitstream. In some embodiments, the first statistical value can be a variance. In some embodiments, the target value of the sample can include one of the following: a first predetermined value or a second statistical value corresponding to the sample. In one example, the first predetermined value can be zero or the second statistical value can be an average value. In some embodiments, the probability can be determined based on statistical information.

[0078] In some embodiments, at least one of the following thresholds may be indicated in the bitstream: a first threshold, a second threshold, a third threshold, a fourth threshold, a fifth threshold, a sixth threshold, a seventh threshold, an eighth threshold, a ninth threshold, a tenth threshold, an eleventh threshold, a twelfth threshold, a thirteenth threshold, or a fourteenth threshold.

[0079] In some embodiments, the second representation may be reconstructed based on the at least one sample at 1506. Furthermore, a conversion may be performed based on the reconstructed second representation.

[0080] In some embodiments, the at least one sample may include all samples of the second representation. In this case, the value of the at least one sample may be determined to reconstruct the second representation.

[0081] In some embodiments, the second representation may include at least one sample and a first set of samples different from the at least one sample. A value of the at least one sample may be determined. The value of the first set of samples may be obtained by entropy decoding the bitstream based on statistical information. The second representation may be reconstructed based on the value of the first set of samples and the determined value of the at least one sample.

[0082] In some embodiments, to determine the value of at least one sample, the value of each sample in the at least one sample can be determined as a second predetermined value or a third statistical value corresponding to the sample. In one example, the second predetermined value can be a constant scalar, such as zero. For example, the third statistical value can be an average value.

[0083] In some embodiments, the first representation may be an implicit representation of the visual data.At 1506, the visual data may be reconstructed by performing a composition transform on the reconstructed second representation.

[0084] In some embodiments, the first representation may be a residual implicit representation of the visual data. At 1506, a quantized implicit representation of the visual data may be generated based on the reconstructed second representation and a fourth statistical value included in the statistical information. The visual data may be reconstructed by performing a synthesis transform on the quantized implicit representation.

[0085] In some embodiments, the at least one sample may include all samples of the second representation.At 1506, a bitstream may be generated by performing an entropy encoding and decoding process on the statistical information.

[0086] In some embodiments, the second representation may include at least one sample and a first set of samples different from the at least one sample. At 1506, the first portion of the bitstream may be generated by performing an entropy encoding and decoding process on the statistical information. The second portion of the bitstream may be generated by performing an entropy encoding and decoding process on the first set of samples based on the statistical information.

[0087] In some embodiments, the first representation may be an implicit representation of the visual data. The method may further include: generating an implicit representation by performing an analytical transformation on the visual data using a first neural network; and obtaining a second representation by quantizing the implicit representation.

[0088] In some embodiments, the first representation may be a residual latent representation of the visual data. The method may further include: generating an implicit representation of the visual data by performing an analytical transformation on the visual data using a first neural network; generating a residual latent representation based on the latent representation and a fourth statistical value included in the statistical information; and obtaining a second representation by quantizing the residual latent representation. In some embodiments, the fourth statistical value may be an average value.

[0089] In some embodiments, the statistical information may be generated using a second neural network. In one example, the second neural network may be an estimation model. The second neural network may include a sub-network based on a neural network. The input of the second neural network may include a bit stream or an implicit representation of the visual data.

[0090] In some embodiments, the statistical information may include a first statistical value and a second statistical value. The second neural network may include a first subnetwork for generating the first statistical value and a second subnetwork for generating the second statistical value. The first statistical value may be a variance, and the second statistical value may be an average value.

[0091] In some embodiments, the first statistic may be a variance of a Gaussian probability distribution and the second statistic may be a mean of the Gaussian probability distribution. Additionally or alternatively, the first subnetwork may be a superscale decoder subnetwork and the second subnetwork may be a superdecoder subnetwork.

[0092] In some embodiments, at least one of the following may be indicated in the bitstream: information about whether to apply the method, or information about how to apply the method. In some embodiments, at least one of the following may depend on the color format and / or color components of the visual data: information about whether to apply the method, or information about how to apply the method.

[0093] In some embodiments, the values ​​included in the bitstream may be encoded and decoded at one of the following: sequence level, picture level, slice level, or block level. In some embodiments, the values ​​included in the bitstream may be binarized before being encoded and decoded. In some embodiments, the values ​​included in the bitstream may be encoded and decoded using at least one arithmetic codec context.

[0094] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of visual data, and the bitstream is generated by a method performed by a visual data processing device. The method includes: for conversion between visual data and a bitstream of visual data, obtaining statistical information associated with a first representation of the visual data, the first representation is generated based on applying a first neural network to the visual data; determining at least one sample from a second representation of the visual data based on the statistical information, the value of the at least one sample is not in the bitstream, the second representation is obtained by quantizing the first representation; and generating a bitstream based on the determination.

[0095] According to some other embodiments of the present disclosure, a method for storing a bitstream of visual data is provided. The method includes: for conversion between visual data and a bitstream of visual data, obtaining statistical information associated with a first representation of the visual data, the first representation being generated based on applying a first neural network to the visual data; determining at least one sample from a second representation of the visual data based on the statistical information, the value of the at least one sample not being in the bitstream, the second representation being obtained by quantizing the first representation; generating a bitstream based on the determination; and storing the bitstream in a non-transitory computer-readable recording medium.

[0096] Embodiments of the present disclosure may be described according to the following clauses, features of which may be combined in any reasonable way.

[0097] Item 1. A method for visual data processing, comprising: for conversion between visual data and a bitstream of the visual data, obtaining statistical information associated with a first representation of the visual data, the first representation being generated based on applying a first neural network to the visual data; determining at least one sample from a second representation of the visual data based on the statistical information, the value of the at least one sample not being in the bitstream, the second representation being obtained by quantizing the first representation; and performing the conversion based on the determination.

[0098] Clause 2. The method of clause 1, wherein the first representation comprises an implicit representation of the visual data or a residual implicit representation of the visual data.

[0099] Clause 3. A method according to any one of clauses 1-2, wherein determining the at least one sample from the second representation includes: determining that the at least one sample includes a sample of the second representation if at least one of the following conditions is determined to be satisfied: a first statistical value corresponding to the sample is less than a first threshold, a value determined based on the first statistical value is less than a second threshold, or the probability that the value of the sample is equal to a target value is less than a third threshold.

[0100] Clause 4. A method according to any one of clauses 1-2, wherein determining the at least one sample from the second representation includes: determining that the at least one sample includes a sample in a block of the second representation if at least one of the following conditions is determined to be satisfied: a first statistical value corresponding to each sample in the block is less than a fourth threshold, a first metric is less than a fifth threshold, the first metric is determined based on the first statistical value corresponding to the sample in the block, a probability that the value of each sample in the block is equal to a target value is less than a sixth threshold, or a second metric is less than a seventh threshold, the second metric is determined based on the probability that the value of the sample in the block is equal to the corresponding target value.

[0101] Clause 5. A method according to any one of clauses 1-2, wherein determining the at least one sample from the second representation includes: determining that the at least one sample includes a sample of the second representation if at least one of the following conditions is determined to be satisfied: a first statistical value corresponding to the sample is greater than an eighth threshold, a value determined based on the first statistical value is greater than a ninth threshold, or a probability that the value of the sample is equal to a target value is greater than a tenth threshold.

[0102] Clause 6. A method according to any one of clauses 1-2, wherein determining the at least one sample from the second representation includes: determining that the at least one sample includes a sample in a block of the second representation if at least one of the following conditions is determined to be satisfied: a first statistical value corresponding to each sample in the block is greater than an eleventh threshold, a first metric is greater than a twelfth threshold, the first metric is determined based on the first statistical value corresponding to the sample in the block, a probability that the value of each sample in the block is equal to a target value is greater than a thirteenth threshold, or a second metric is greater than a fourteenth threshold, the second metric is determined based on the probability that the value of the sample in the block is equal to the corresponding target value.

[0103] Clause 7. A method according to clause 4 or 6, wherein the first metric is the average of the first statistical values, the minimum first statistical value among the first statistical values, or the maximum first statistical value among the first statistical values, or the second metric is the average of the probabilities, the minimum probability among the probabilities, or the maximum probability among the probabilities.

[0104] Clause 8. The method of any one of clauses 4 and 7-8, wherein the size of the block is N times M, and both N and M are positive integers.

[0105] Clause 9. The method of clause 8, wherein N and M are indicated in the bitstream.

[0106] Clause 10. The method of any one of clauses 3-9, wherein the first statistic is a variance.

[0107] Clause 11. A method according to any one of clauses 3-10, wherein the target value of the sample comprises one of: a first predetermined value, or a second statistical value corresponding to the sample.

[0108] Clause 12. The method of clause 11, wherein the first predetermined value is zero, or the second statistical value is an average value.

[0109] Clause 13. The method of any of clauses 3-12, wherein the probability is determined based on the statistical information.

[0110] Clause 14. A method according to any one of clauses 3-13, wherein at least one of the following thresholds is indicated in the bitstream: the first threshold, the second threshold, the third threshold, the fourth threshold, the fifth threshold, the sixth threshold, the seventh threshold, the eighth threshold, the ninth threshold, the tenth threshold, the eleventh threshold, the twelfth threshold, the thirteenth threshold or the fourteenth threshold.

[0111] Clause 15. The method of any of clauses 1-14, wherein performing the conversion comprises: reconstructing the second representation based on the at least one sample; and performing the conversion based on the reconstructed second representation.

[0112] Clause 16. The method of clause 15, wherein the at least one sample comprises all samples of the second representation, and reconstructing the second representation comprises determining a value of the at least one sample to reconstruct the second representation.

[0113] Clause 17. A method according to clause 15, wherein the second representation includes the at least one sample and a first group of samples different from the at least one sample, and reconstructing the second representation includes: determining a value of the at least one sample; obtaining the value of the first group of samples by performing entropy decoding processing on the bitstream based on the statistical information; and reconstructing the second representation based on the value of the first group of samples and the determined value of the at least one sample.

[0114] Clause 18. The method of any one of clauses 16-17, wherein determining the value of the at least one sample comprises determining the value of each sample of the at least one sample as a second predetermined value or a third statistical value corresponding to the sample.

[0115] Clause 19. The method of Clause 18, wherein the second predetermined value is a constant scalar.

[0116] Clause 20. The method of clause 18, wherein the second predetermined value is zero.

[0117] Clause 21. The method according to any one of clauses 18-20, wherein the third statistical value is a mean value.

[0118] Clause 22. A method according to any one of clauses 15-21, wherein the first representation is an implicit representation of the visual data, and performing the conversion based on the reconstructed second representation includes: reconstructing the visual data by performing a synthetic transformation on the reconstructed second representation.

[0119] Clause 23. A method according to any one of clauses 15-21, wherein the first representation is a residual implicit representation of the visual data, and performing the conversion comprises: generating a quantized implicit representation of the visual data based on the reconstructed second representation and a fourth statistical value included in the statistical information; and reconstructing the visual data by performing a synthetic transformation on the quantized implicit representation.

[0120] Clause 24. The method of any of clauses 1-14, wherein the at least one sample comprises all samples of the second representation, and performing the conversion comprises generating the bitstream by performing an entropy encoding process on the statistical information.

[0121] Clause 25. A method according to any one of clauses 1-14, wherein the second representation includes the at least one sample and a first group of samples different from the at least one sample, and performing the conversion includes: generating a first part of the bitstream by performing an entropy encoding process on the statistical information; and generating a second part of the bitstream by performing an entropy encoding process on the first group of samples based on the statistical information.

[0122] Clause 26. A method according to any one of clauses 24-25, wherein the first representation is an implicit representation of the visual data, and the method further comprises: generating the implicit representation by performing an analytical transformation on the visual data using the first neural network; and obtaining the second representation by quantizing the implicit representation.

[0123] Clause 27. A method according to any one of Clauses 24-25, wherein the first representation is a residual latent representation of the visual data, and the method further comprises: generating an latent representation of the visual data by performing an analytical transformation on the visual data using the first neural network; generating the residual latent representation based on the latent representation and a fourth statistical value included in the statistical information; and obtaining the second representation by quantizing the residual latent representation.

[0124] Clause 28. The method of clause 23 or 28, wherein the fourth statistical value is an average value.

[0125] Clause 29. The method of any one of clauses 1-28, wherein the statistical information is generated using a second neural network.

[0126] Clause 30. The method of clause 29, wherein the second neural network is an estimation model.

[0127] Clause 31. A method according to any of clauses 29-30, wherein the second neural network comprises a sub-network based on a neural network.

[0128] Clause 32. A method according to any of clauses 29-31, wherein the input to the second neural network comprises an implicit representation of the bitstream or the visual data.

[0129] Clause 33. A method according to any of clauses 29-30, wherein the statistical information includes a first statistical value and a second statistical value, and the second neural network includes a first subnetwork for generating the first statistical value and a second subnetwork for generating the second statistical value.

[0130] Clause 34. The method of clause 33, wherein the first statistic is a variance and the second statistic is a mean.

[0131] Clause 35. The method of clause 33, wherein the first statistic is a variance of a Gaussian probability distribution and the second statistic is a mean of the Gaussian probability distribution.

[0132] Clause 36. The method of any of clauses 33-35, wherein the first sub-network is a superscale decoder sub-network and the second sub-network is a superdecoder sub-network.

[0133] Clause 37. A method according to any of clauses 1-36, wherein at least one of the following is indicated in the bitstream: information on whether to apply the method, or information on how to apply the method.

[0134] Clause 38. A method according to any of clauses 1-36, wherein at least one of the following depends on the color format and / or color components of the visual data: information about whether to apply the method, or information about how to apply the method.

[0135] Clause 39. A method according to any of clauses 1-38, wherein the values ​​included in the bitstream are encoded at one of: sequence level, picture level, slice level, or block level.

[0136] Clause 40. The method of any of clauses 1-39, wherein values ​​included in the bitstream are binarized prior to being encoded or decoded.

[0137] Clause 41. The method of any of clauses 1-40, wherein values ​​included in the bitstream are encoded using at least one arithmetic coding context.

[0138] Clause 42. A method according to any of clauses 1-41, wherein the visual data comprises a picture or image of a video.

[0139] Clause 43. The method of any one of clauses 1-42, wherein the converting comprises encoding the visual data into the bitstream.

[0140] Clause 44. The method of any of clauses 1-42, wherein the converting comprises decoding the visual data from the bitstream.

[0141] Clause 45. An apparatus for visual data processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of clauses 1-44.

[0142] Clause 46. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method of any of clauses 1-44.

[0143] Item 47. A non-transitory computer-readable recording medium storing a bitstream of visual data, the bitstream being generated by a method performed by a visual data processing device, wherein the method comprises: obtaining statistical information associated with a first representation of the visual data for conversion between visual data and a bitstream of the visual data, the first representation being generated based on applying a first neural network to the visual data; determining at least one sample from a second representation of the visual data based on the statistical information, the value of the at least one sample being not in the bitstream, the second representation being obtained by quantizing the first representation; and generating the bitstream based on the determination.

[0144] Clause 48. A method for storing a bitstream of visual data, comprising: obtaining statistical information associated with a first representation of the visual data for conversion between visual data and a bitstream of the visual data, the first representation being generated based on applying a first neural network to the visual data; determining at least one sample from a second representation of the visual data based on the statistical information, the value of the at least one sample not being in the bitstream, the second representation being obtained by quantizing the first representation; generating the bitstream based on the determination; and storing the bitstream in a non-transitory computer-readable recording medium. Example Device

[0145] Fig.16 A block diagram of a computing device 1600 is shown in which various embodiments of the present disclosure may be implemented. The computing device 1600 may be implemented as or included in the source device 110 (or visual data encoder 114) or the destination device 120 (or visual data decoder 124).

[0146] It should be understood that Fig.16 The computing device 1600 shown in FIG. 1 is for illustrative purposes only and is not intended to in any way imply any limitation on the functionality and scope of the embodiments of the present disclosure.

[0147] like Fig.16 As shown, computing device 1600 includes a general computing device 1600. Computing device 1600 may include at least one or more processors or processing units 1610, memory 1620, storage unit 1630, one or more communication units 1640, one or more input devices 1650, and one or more output devices 1660.

[0148] In some embodiments, computing device 1600 can be implemented as any user terminal or server terminal with computing power. The server terminal can be a server provided by a service provider, a large computing device, etc. The user terminal can be, for example, any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet computer, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device, or any combination thereof, and includes the accessories and peripherals of these devices, or any combination thereof. It is conceivable that computing device 1600 can support any type of interface to the user (such as a "wearable" circuit device, etc.).

[0149] Processing unit 1610 may be a physical processor or a virtual processor and may implement various processes based on a program stored in memory 1620. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of computing device 1600. Processing unit 1610 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0150] The computing device 1600 typically includes various computer storage media. Such media can be any media accessible by the computing device 1600, including but not limited to volatile media and non-volatile media, or removable media and non-removable media. The memory 1620 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM) or flash memory), or any combination thereof. The storage unit 1630 can be any removable or non-removable medium, and can include machine-readable media, such as a memory, a flash drive, a disk, or other media that can be used to store information and / or visual data and can be accessed in the computing device 1600.

[0151] The computing device 1600 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Fig.16 It is not shown in the figure, but a disk drive for reading from and / or writing to a removable non-volatile disk and an optical drive for reading from and / or writing to a removable non-volatile optical disk may be provided. In this case, each drive may be connected to the bus (not shown) via one or more visual data medium interfaces.

[0152] The communication unit 1640 communicates with another computing device via a communication medium. In addition, the functions of the components in the computing device 1600 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 1600 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general network nodes.

[0153] Input device 1650 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 1660 may be one or more of various output devices, such as a display, speaker, printer, etc. With the aid of communication unit 1640, computing device 1600 may also communicate with one or more external devices (not shown), such as storage devices and display devices, and computing device 1600 may also communicate with one or more devices that enable a user to interact with computing device 1600, or any device that enables computing device 1600 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.), if necessary. Such communication may be performed via an input / output (I / O) interface (not shown).

[0154] In some embodiments, some or all components of the computing device 1600 may also be arranged in a cloud computing architecture rather than being integrated in a single device. In a cloud computing architecture, components may be provided remotely and work together to implement the functions described in the present disclosure. In some embodiments, cloud computing provides computing, software, visual data access and storage services, which will not require the end user to know the physical location or configuration of the system or hardware that provides these services. In various embodiments, cloud computing provides services via a wide area network (e.g., the Internet) using a suitable protocol. For example, a cloud computing provider provides applications via a wide area network, which can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding visual data may be stored on a remote server. The computing resources in a cloud computing environment may be merged or distributed at the location of a remote visual data center. Cloud computing infrastructures may provide services through a shared visual data center, although they appear as a single access point for users. Therefore, cloud computing architectures may be used to provide components and functions described herein from a service provider at a remote location. Alternatively, they may be provided by a conventional server, or installed directly or otherwise on a client device.

[0155] The computing device 1600 can be used to implement the visual data encoding / decoding in the embodiments of the present disclosure, and the memory 1620 can include one or more visual data encoding / decoding modules 1625 having one or more program instructions. These modules can be accessed and executed by the processing unit 1610 to perform the functions of the various embodiments described herein.

[0156] In an example embodiment where visual data encoding is performed, input device 1650 may receive visual data as input to be encoded 1650. The visual data may be processed by, for example, visual data encoding / decoding module 1625 to generate an encoded bitstream. The encoded bitstream may be provided as output 1680 via output device 1660.

[0157] In an example embodiment where visual data decoding is performed, input device 1650 may receive an encoded bitstream as input 1650. The encoded bitstream may be processed by, for example, visual data codec module 1625 to generate decoded visual data. The decoded visual data may be provided as output 1680 via output device 1660.

[0158] Although the present disclosure has been specifically shown and described with reference to the preferred embodiments of the present disclosure, it will be understood by those skilled in the art that various changes can be made in form and detail without departing from the spirit and scope of the present application as defined by the appended claims. These changes are intended to be encompassed by the scope of the present application. Therefore, the foregoing description of the embodiments of the present application is not intended to be limiting.

Claims

1. A method for visual data processing, comprising: For conversion between visual data and a bitstream of the visual data, obtaining statistical information associated with a first representation of the visual data, the first representation being generated based on applying a first neural network to the visual data; determining at least one sample from a second representation of the visual data based on the statistical information, the value of the at least one sample not being in the bitstream, the second representation being obtained by quantizing the first representation; as well as The converting is performed based on the determination.

2. The method of claim 1, wherein the first representation comprises an implicit representation of the visual data or a residual implicit representation of the visual data.

3. The method according to any one of claims 1-2, wherein determining the at least one sample from the second representation comprises: The at least one sample is determined to include a sample of the second representation if at least one of the following conditions is determined to be satisfied: A first statistical value corresponding to the sample is less than a first threshold, The value determined based on the first statistical value is less than a second threshold, or The probability that the value of the sample is equal to the target value is less than a third threshold.

4. The method according to any one of claims 1-2, wherein determining the at least one sample from the second representation comprises: The at least one sample is determined to include a sample in a block of the second representation if at least one of the following conditions is determined to be satisfied: a first statistical value corresponding to each sample in the block is less than a fourth threshold, a first metric is less than a fifth threshold, the first metric being determined based on a first statistic corresponding to samples in the block, The probability that the value of each sample in the block is equal to the target value is less than a sixth threshold, or The second metric is less than a seventh threshold, the second metric being determined based on a probability that the values ​​of the samples in the block are equal to corresponding target values.

5. The method according to any one of claims 1-2, wherein determining the at least one sample from the second representation comprises: The at least one sample is determined to include a sample of the second representation if at least one of the following conditions is determined to be satisfied: A first statistical value corresponding to the sample is greater than an eighth threshold, The value determined based on the first statistical value is greater than a ninth threshold, or The probability that the value of the sample is equal to the target value is greater than a tenth threshold.

6. The method according to any one of claims 1-2, wherein determining the at least one sample from the second representation comprises: The at least one sample is determined to include a sample in a block of the second representation if at least one of the following conditions is determined to be satisfied: a first statistical value corresponding to each sample in the block is greater than an eleventh threshold, a first metric greater than a twelfth threshold, the first metric being determined based on a first statistic corresponding to samples in the block, The probability that the value of each sample in the block is equal to the target value is greater than a thirteenth threshold, or A second metric is greater than a fourteenth threshold, the second metric being determined based on a probability that values ​​of samples in the block are equal to corresponding target values.

7. The method according to claim 4 or 6, wherein the first metric is an average of the first statistical values, a minimum first statistical value among the first statistical values, or a maximum first statistical value among the first statistical values, or The second metric is an average of the probabilities, a minimum probability among the probabilities, or a maximum probability among the probabilities.

8. The method according to any one of claims 4 and 7-8, wherein the size of the block is N times M, and both N and M are positive integers.

9. The method of claim 8, wherein N and M are indicated in the bitstream.

10. The method according to any one of claims 3 to 9, wherein the first statistical value is a variance.

11. The method according to any one of claims 3 to 10, wherein the target value of the sample comprises one of the following: a first predetermined value, or A second statistical value corresponding to the sample. The method according to claim 11 , wherein the first predetermined value is zero, or the second statistical value is an average value.

13. The method according to any one of claims 3 to 12, wherein the probability is determined based on the statistical information.

14. The method according to any one of claims 3-13, wherein at least one of the following thresholds is indicated in the bitstream: the first threshold, the second threshold, the third threshold, the fourth threshold, the fifth threshold, the sixth threshold, the seventh threshold, the eighth threshold, the ninth threshold, the tenth threshold, the eleventh threshold, the twelfth threshold, the thirteenth threshold or the fourteenth threshold.

15. The method according to any one of claims 1 to 14, wherein performing the conversion comprises: reconstructing the second representation based on the at least one sample; as well as The conversion is performed based on the reconstructed second representation.

16. The method of claim 15, wherein the at least one sample comprises all samples of the second representation, and reconstructing the second representation comprises: A value of the at least one sample is determined to reconstruct the second representation.

17. The method of claim 15, wherein the second representation comprises the at least one sample and a first set of samples different from the at least one sample, and reconstructing the second representation comprises: determining a value of the at least one sample; Obtaining values ​​of the first set of samples by performing an entropy decoding process on the bitstream based on the statistical information; as well as The second representation is reconstructed based on the values ​​of the first set of samples and the determined value of the at least one sample.

18. The method of any one of claims 16-17, wherein determining the value of the at least one sample comprises: The value of each sample of the at least one sample is determined as a second predetermined value or a third statistical value corresponding to the sample. The method of claim 18 , wherein the second predetermined value is a constant scalar.

20. The method of claim 18, wherein the second predetermined value is zero.

21. The method according to any one of claims 18 to 20, wherein the third statistical value is a mean value.

22. The method of any one of claims 15-21, wherein the first representation is an implicit representation of the visual data, and performing the conversion based on the reconstructed second representation comprises: The visual data is reconstructed by performing a synthetic transform on the reconstructed second representation.

23. The method of any one of claims 15-21, wherein the first representation is a residual latent representation of the visual data, and performing the conversion comprises: generating a quantized latent representation of the visual data based on the reconstructed second representation and a fourth statistical value included in the statistical information; as well as The visual data is reconstructed by performing a synthetic transform on the quantized latent representation.

24. The method of any one of claims 1-14, wherein the at least one sample comprises all samples of the second representation, and performing the conversion comprises: The bitstream is generated by performing an entropy encoding process on the statistical information.

25. The method of any one of claims 1-14, wherein the second representation comprises the at least one sample and a first set of samples different from the at least one sample, and performing the conversion comprises: generating a first portion of the bitstream by performing an entropy encoding process on the statistical information; as well as A second portion of the bitstream is generated by performing an entropy encoding process on the first set of samples based on the statistical information.

26. A method according to any one of claims 24-25, wherein the first representation is an implicit representation of the visual data, and the method further comprises: generating the latent representation by performing an analytical transformation on the visual data using the first neural network; as well as The second representation is obtained by quantizing the latent representation.

27. The method according to any one of claims 24-25, wherein the first representation is a residual latent representation of the visual data, and the method further comprises: generating a latent representation of the visual data by performing an analytical transformation on the visual data using the first neural network; generating the residual latent representation based on the latent representation and a fourth statistical value included in the statistical information; and The second representation is obtained by quantizing the residual latent representation.

28. The method according to claim 23 or 28, wherein the fourth statistical value is a mean value.

29. The method of any one of claims 1-28, wherein the statistical information is generated using a second neural network.

30. The method of claim 29, wherein the second neural network is an estimation model.

31. The method of any one of claims 29-30, wherein the second neural network comprises a sub-network based on a neural network.

32. A method according to any one of claims 29-31, wherein the input of the second neural network comprises an implicit representation of the bit stream or the visual data.

33. The method according to any one of claims 29-30, wherein the statistical information includes a first statistical value and a second statistical value, and the second neural network includes a first sub-network for generating the first statistical value and a second sub-network for generating the second statistical value.

34. The method of claim 33, wherein the first statistic is a variance and the second statistic is a mean.

35. The method of claim 33, wherein the first statistic is a variance of a Gaussian probability distribution and the second statistic is a mean of the Gaussian probability distribution.

36. The method of any one of claims 33-35, wherein the first sub-network is a superscale decoder sub-network and the second sub-network is a superdecoder sub-network.

37. The method of any one of claims 1-36, wherein at least one of the following is indicated in the bitstream: information on whether the method was applied, or Information on how to apply the method.

38. The method of any one of claims 1-36, wherein at least one of the following depends on the color format and / or color components of the visual data: information on whether the method was applied, or Information on how to apply the method.

39. The method of any one of claims 1-38, wherein a value included in the bitstream is encoded or decoded at one of: Sequence level, Picture level, Stripe level, or Block level.

40. The method according to any one of claims 1-39, wherein the values ​​included in the bitstream are binarized before being encoded and decoded.

41. The method according to any one of claims 1-40, wherein values ​​included in the bitstream are encoded using at least one arithmetic coding context.

42. The method of any one of claims 1-41, wherein the visual data comprises a picture or image of a video.

43. The method of any one of claims 1-42, wherein the converting comprises encoding the visual data into the bitstream.

44. The method of any one of claims 1-42, wherein the converting comprises decoding the visual data from the bitstream.

45. An apparatus for visual data processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1-44.

46. ​​A non-transitory computer-readable storage medium storing instructions, the instructions causing a processor to execute the method according to any one of claims 1-44.

47. A non-transitory computer-readable recording medium storing a bit stream of visual data, the bit stream being generated by a method performed by a visual data processing device, wherein the method comprises: For conversion between visual data and a bitstream of the visual data, obtaining statistical information associated with a first representation of the visual data, the first representation being generated based on applying a first neural network to the visual data; determining at least one sample from a second representation of the visual data based on the statistical information, the value of the at least one sample not being in the bitstream, the second representation being obtained by quantizing the first representation; as well as The bitstream is generated based on the determination.

48. A method for storing a bitstream of visual data, comprising: For conversion between visual data and a bitstream of the visual data, obtaining statistical information associated with a first representation of the visual data, the first representation being generated based on applying a first neural network to the visual data; determining at least one sample from a second representation of the visual data based on the statistical information, the value of the at least one sample not being in the bitstream, the second representation being obtained by quantizing the first representation; Based on the determination, generating the bitstream; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

Images