Visual data processing method, device and medium

Abstract

Embodiments of the present disclosure provide a solution for visual data processing. A method for visual data processing is presented. The method includes determining, for a conversion between at least one bitstream of visual data and the visual data, a residual representation of the visual data based at least on a gain parameter and a first probability distribution parameter of the visual data, the residual representation representing a residual value compared to a second probability distribution representation of the visual data, the gain parameter adjusts the value range represented by the residual error; and performing a conversion based on the residual representation.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 369,219, filed on July 22, 2022, which has been assigned to the assignee of the present application and is incorporated herein by reference in its entirety. Technical Field

[0003] Embodiments of the present disclosure relate generally to visual data processing techniques, and more particularly to rate adjustment for visual data processing. Background Art

[0004] Image / video compression is an important technology to reduce the cost of image / video transmission and storage in a lossless or lossy manner. Image / video compression technology can be divided into two branches, classical video codec methods and neural network-based video compression methods. Classical video codec schemes adopt transform-based solutions, in which researchers exploit statistical dependencies in latent variables (e.g., wavelet coefficients) by carefully manually designing entropy codes to model dependencies in the quantization domain. There are two forms of neural network-based video compression: neural network-based codec tools and end-to-end neural network-based video compression. Neural network-based codec tools are embedded into existing classical video codecs as coding tools and are only used as part of the framework, while end-to-end neural network-based video compression is a separate framework developed based on neural networks and does not rely on classical video codecs. The coding and decoding efficiency of image / video codecs is generally expected to be further improved. Summary of the invention

[0005] Embodiments of the present disclosure provide a solution for visual data processing.

[0006] In a first aspect, a method for visual data processing is provided. The method comprises: for conversion between at least one bitstream of visual data and visual data, determining a residual representation of the visual data based on at least a gain parameter and a first probability distribution parameter of the visual data, the residual representation representing a residual value compared to a second probability distribution representation of the visual data, the gain parameter adjusting a value range of the residual representation; and performing the conversion based on the residual representation.

[0007] The method according to the first aspect of the present disclosure adjusts the output rate of the conversion by using a gain parameter. For example, the rate of the bit stream or the rate of the visual data can be adjusted. In this way, the coding efficiency and coding effectiveness can be improved.

[0008] In a second aspect, a visual data processing device is provided. The device includes a processor and a non-volatile memory having instructions thereon. These instructions, when executed by the processor, cause the processor to perform the method according to the first aspect of the present disclosure.

[0009] In a third aspect, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions for causing a processor to execute the method according to the first aspect of the present disclosure.

[0010] In a fourth aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores at least one bitstream of visual data, the at least one bitstream of visual data being generated by a method performed by a visual data processing device. The method comprises: determining a residual representation of the visual data based on at least a gain parameter and a first probability distribution parameter of the visual data, the residual representation representing a residual value compared to a second probability distribution representation of the visual data, the gain parameter adjusting a value range of the residual representation; and generating at least one bitstream based on the residual representation.

[0011] In a fifth aspect, a method for storing at least one bitstream of visual data is provided. The method comprises: determining a residual representation of the visual data based on at least a gain parameter and a first probability distribution parameter of the visual data, the residual representation representing a residual value compared to a second probability distribution representation of the visual data, the gain parameter adjusting a value range of the residual representation; generating at least one bitstream based on the residual representation; and storing the at least one bitstream in a non-transitory computer-readable recording medium.

[0012] This Summary is intended to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become more apparent through the following detailed description with reference to the accompanying drawings. In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0014] Figure 1 A block diagram showing an example visual data encoding and decoding system according to some embodiments of the present disclosure is shown;

[0015] Figure 2 A typical transform coding scheme is shown;

[0016] Figure 3 Images from the Kodak dataset and different representations of the images are shown;

[0017] Figure 4 The network architecture of the autoencoder implementing the hyper-prior model is shown;

[0018] Figure 5A block diagram of the combined model is shown;

[0019] Figure 6 shows the encoding process according to the joint autoregressive and hierarchical priors for learning image compression;

[0020] Figure 7 The decoding processes corresponding to the joint autoregressive and hierarchical priors for learning image compression are shown;

[0021] Figure 8 shows a decoding process according to an embodiment of the present disclosure;

[0022] Fig. 9 A flowchart showing a method for visual data processing according to an embodiment of the present disclosure; and

[0023] Fig.10 A block diagram of a computing device is shown in which various embodiments of the present disclosure may be implemented.

[0024] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. DETAILED DESCRIPTION

[0025] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described only for the purpose of illustrating and helping those skilled in the art to understand and implement the present disclosure, without implying any limitation on the scope of the present disclosure. In addition to the methods described below, the disclosure described herein can also be implemented in various ways.

[0026] In the following description and claims, unless otherwise defined, all scientific and technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs.

[0027] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," and the like indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment must include the particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example embodiment, whether or not explicitly described, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in relation to other embodiments.

[0028] It should be understood that although the terms "first" and "second" etc. can be used to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish one element from another element. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0029] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms "including", "comprising", and / or "having" when used herein indicate the presence of the features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0030] Example Environment

[0031] Figure 1 1 is a block diagram illustrating an example visual data encoding and decoding system 100 that can utilize the techniques of the present disclosure. As shown, the visual data encoding and decoding system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a data encoding device or a visual data encoding device, and the destination device 120 may also be referred to as a data decoding device or a visual data decoding device. In operation, the source device 110 may be configured to generate encoded visual data, and the destination device 120 may be configured to decode the encoded visual data generated by the source device 110. The source device 110 may include a data source 112, a data encoder 114, and an input / output (I / O) interface 116.

[0032] Data sources 112 may include sources such as data capture devices. Examples of data capture devices include, but are not limited to, interfaces that receive data from data providers, computer graphics systems for generating data, and / or combinations thereof.

[0033] The data may include one or more images or one or more pictures of a video. The data encoder 114 encodes the data from the data source 112 to generate a bitstream. The bitstream may include a sequence of bits that form a codec representation of the data. The bitstream may include a codec picture and associated data. The codec picture is a codec representation of the picture. The associated data may include a sequence parameter set, a picture parameter set, and other grammatical structures. The I / O interface 116 may include a modulator / demodulator and / or a transmitter. The encoded data may be directly transmitted to the destination device 120 via the network 130A via the I / O interface 116. The encoded data may also be stored on a storage medium / server 130B for access by the destination device 120.

[0034] The destination device 120 may include an I / O interface 126, a data decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain encoded data from the source device 110 or the storage medium / server 130B. The data decoder 124 may decode the encoded data. The display device 122 may display the decoded data to the user. The display device 122 may be integrated with the destination device 120, or may be outside the destination device 120, which is configured to be connected to an external display device interface.

[0035] The data encoder 114 and the data decoder 124 may operate according to a data codec standard, such as a video codec standard or a still picture codec standard and other existing and / or further standards.

[0036] Some example embodiments of the present disclosure will be described in detail below. It should be noted that the section titles used in this document are for ease of understanding, and the embodiments disclosed in the section are not limited to that section. In addition, although some embodiments are described with reference to general video codecs or other specific data codecs, the disclosed technology is also applicable to other codec technologies. In addition, although some embodiments describe the coding and decoding steps in detail, it should be understood that the corresponding decoding steps of canceling the coding will be implemented by the decoder. In addition, the term data processing includes data coding or compression, data decoding or decompression, and data transcoding, in which data is represented from one compression format to another compression format or at different compression bit rates.

[0037] More details of embodiments of the present disclosure related to rate adjustment for visual data processing will be described below.As used herein, the term "data" or "visual data" may refer to an image, a frame in a video, a picture in a video, a video, or any other data suitable for being encoded or decoded.

[0038] 1. Brief Overview

[0039] A neural network-based image and video compression method includes an autoregressive subnetwork and an entropy coding and decoding engine, wherein the entropy coding and decoding are performed independently of the autoregressive subnetwork. A Gained Variational Autoencoder (G-VAE) is used for continuous rate adaptation.

[0040] 2. Introduction

[0041] The past decade has witnessed the rapid development of deep learning in various fields, especially in computer vision and image processing. Inspired by the great success of deep learning techniques in the field of computer vision, many researchers have shifted their attention from traditional image / video compression techniques to neural image / video compression techniques. Neural networks were originally invented through interdisciplinary research in neuroscience and mathematics. It has shown strong capabilities in the context of nonlinear transformation and classification. Image / video compression techniques based on neural networks have made significant progress during the past five years. It is reported that the latest neural network-based image compression algorithms achieve RD performance comparable to the latest video codec standard - Versatile Video Coding (VVC) - developed by the Joint Video Experts Group (JVET) with experts from MPEG and VCEG. With the continuous improvement of the performance of neural image compression, neural network-based video compression has become an actively developed research area. However, due to the inherent difficulty of the problem, neural network-based video codecs still remain in their infancy.

[0042] 2.1 Image / Video Compression

[0043] Image / video compression generally refers to computing techniques that compress images / videos into binary codes for easy storage and transmission. The binary codes may or may not support lossless reconstruction of the original image / video, which is called lossless compression and lossy compression. Most research focuses on lossy compression because lossless reconstruction is not necessary in most scenarios. Usually, the performance of image / video compression algorithms is evaluated from two aspects, namely compression ratio and reconstruction quality. The compression ratio is directly related to the number of binary codes, the smaller the better; the reconstruction quality is measured by comparing the reconstructed image / video with the original image / video, the higher the better.

[0044] Image / video compression technology can be divided into two branches, classical video codec methods and neural network-based video compression methods. Classical video codec schemes adopt transform-based solutions, in which researchers exploit statistical dependencies in latent variables (e.g., wavelet coefficients) by carefully manually designing entropy codes to model dependencies in the quantization domain. There are two forms of neural network-based video compression: neural network-based codec tools and end-to-end neural network-based video compression. The former is embedded into existing classical video codecs as a coding tool and is only used as part of the framework, while the latter is a separate framework developed based on neural networks and does not rely on classical video codecs.

[0045] Over the past three decades, a series of classic video codec standards have been developed to accommodate the increased visual content. The international standards organization ISO / IEC has two expert groups, the Joint Photographic Experts Group (JPEG) and the Moving Picture Experts Group (MPEG), and ITU-T also has its own Video Codec Experts Group (VCEG), which is used for the standardization of image / video codec technologies. Influential video codec standards released by these organizations include JPEG, JPEG 2000, H.262, H.264 / AVC, and H.265 / HEVC. After H.265 / HEVC, the Joint Video Experts Group (JVET) formed by MPEG and VCEG has been working on a new video codec standard, the Versatile Video Codec (VVC). The first version of VVC was released in July 2020. Compared with HEVC, VVC is reported to have an average bit rate reduction of 50% at the same visual quality.

[0046] Neural network-based image / video compression is not a new solution, as many researchers have been working on neural network-based image codecs. However, the network architectures in the past were relatively shallow and the performance was not satisfactory. Benefiting from the support of large amounts of data and powerful computing resources, neural network-based methods have been better utilized in various applications. Currently, neural network-based image / video compression has shown encouraging progress, proving its feasibility. However, the technology is still far from mature and many challenges need to be addressed.

[0047] 2.2 Neural Networks

[0048] A neural network (also called an artificial neural network (ANN)) is a computational model used in machine learning techniques that is typically composed of multiple processing layers, and each layer is composed of multiple simple but nonlinear basic computational units. One benefit of such deep networks is believed to be the ability to process data with multiple levels of abstraction and convert the data into different kinds of representations. Note that these representations are not manually designed; instead, deep networks include processing layers that learn from massive amounts of data using general machine learning procedures. Deep learning eliminates the necessity for manual representations and is therefore believed to be useful, particularly for processing native unstructured data such as acoustic and visual signals, while processing such data has been a long-standing difficulty in the field of artificial intelligence.

[0049] 2.3 Neural Networks for Image Compression

[0050] Existing neural network approaches for image compression can be divided into two categories, namely pixel probability modeling and autoencoders. The former belongs to predictive encoding and decoding strategies, while the latter is a transform-based solution. Sometimes, these two approaches are combined in the literature.

[0051] 2.3.1 Pixel Probabilistic Modeling

[0052] According to Shannon information theory, the optimal method for lossless encoding and decoding can achieve the minimum encoding and decoding ratio log 2 p(x), where p(x) is the probability of symbol x. Many lossless codecs have been developed in the literature, and among them arithmetic coding is considered the best. Given a probability distribution p(x), arithmetic coding ensures that the codec ratio is as close as possible to its theoretical limit log without considering round-off errors. 2 p(x). Therefore, the remaining question is how to determine the probability, which is very challenging for natural images / videos due to the dimensionality reduction.

[0053] Following the predictive coding strategy, one way to model p(x) is to predict pixel-by-pixel probability in raster scan order based on previous observations, where x is the image.

[0054] p(x)=p(x 1 )p(x 2 |x 1 )…p(x i |x 1 , …, x i-1 )…p(x m×n |x 1 , …, x m×n-1 ) (1)

[0055] Where m and n are the height and width of the image, respectively. The previous observation is also called the context of the current pixel. When the image is large, it may be difficult to estimate the conditional probability, so a simplified approach is to limit the scope of its context.

[0056] p(x)=p(x 1 )p(x 2 |x 1 )…p(x i |x i-k , …, x i-1 )…p(x m×n |x m×n-k , …, x m×n-1 ) (2)

[0057] Where k is a predefined constant that controls the scope of the context.

[0058] It should be noted that the condition may also consider sample values ​​of other color components. For example, when encoding and decoding RGB color components, the R sample depends on the previously encoded pixel (including R / G / B samples), and the current G sample can be encoded and decoded based on the previously encoded pixel and the current R sample pair, while for encoding and decoding the current B sample, the previously encoded pixel and the current R sample and the current G sample may also be considered.

[0059] Neural networks were initially introduced for computer vision tasks and have been shown to be effective in regression and classification problems. Therefore, it has been proposed to use neural networks to estimate a given its context x. 1 , x 2 , ..., x i-1 The probability p(x i ). The pixel probability is proposed for binary images, i.e., x i ∈{-1,+1}. A neural autoregressive distribution estimator (NADF) is designed for pixel-wise probability modeling, where is a feed-forward network with a single hidden layer. Similar work is presented, where the feed-forward network also has skip connections to the hidden layer and parameters are also shared. Experiments are performed on the binarized MNIST dataset. NADF is extended to the real-valued model RNADE, where the probability p(x i |x 1 ,…,x i-1 ) is derived by a mixture of Gaussians. Their feed-forward networks also have a single hidden layer, but the hidden layer has rescaling to avoid saturation and uses rectified linear units (ReLU) instead of sigmoids. NADF and RNADE are improved by reorganizing the order of pixels and by deeper neural networks.

[0060] Designing advanced neural networks plays an important role in improving pixel probability modeling. Multidimensional long short-term memory (LSTM) is proposed, which works with a mixture of conditional Gaussian scale mixtures for probabilistic modeling. LSTM is a special kind of recurrent neural network (RNN), and it is proven to be good at modeling sequence data. The spatial variant of LSTM was later used for images. Several different neural networks were studied, including RNN and CNN, namely PixelRNN and PixelCNN. In PixelRNN, two variants of LSTM are proposed, called row LSTM and diagonal BiLSTM, the latter of which is specifically designed for images. PixelRNN synthesizes residual connections to help train deep neural networks with up to 12 layers. In PixelCNN, masked convolutions are used to adapt to the shape of the context. Compared with previous work, PixelRNN and PixelCNN are more specialized for natural images: they consider pixels as discrete values ​​(e.g., 0, 1, ..., 255) and predict a multinomial distribution of discrete values; they process color images in RGB color space; they work well on the large-scale image dataset ImageNet. GatedPixelCNN is proposed to improve PixelCNN and achieves comparable performance to PixelRNN but with much lower complexity. PixelCNN++ is proposed on PixelCNN with the following improvements: discretized logistic mixture likelihoods are used instead of 256-way multinomial distributions; downsampling is used to capture structures at multiple resolutions; additional shortcut connections are introduced to speed up training; dropout is adopted for regularization; RGB is combined for one pixel. PixelSNAIL is proposed, where random convolutions are combined with self-attention.

[0061] Most of the above methods directly model the probability distribution in the pixel domain. Some researchers have also tried to model the probability distribution as a condition of an explicit or implicit representation. That is, we can estimate:

[0062]

[0063] Where h is an additional condition, and p(x)=p(h)p(x|h) means that the modeling is split into an unconditional one and a conditional one. The additional condition can be image label information or a high-level representation.

[0064] 2.3.2 Autoencoder

[0065] An autoencoder derived from a well-known work of is proposed. The method is trained for dimensionality reduction and contains two parts: encoding and decoding. The encoding part converts the high-dimensional input signal into a low-dimensional representation, usually with reduced spatial size but with a larger number of channels. The decoding part tries to recover the high-dimensional input from the low-dimensional representation. Autoencoders enable automatic learning of representations and eliminate the need for hand-crafted features, which is also considered one of the most important advantages of neural networks.

[0066] Figure 2 A typical transform coding scheme 200 is shown. The original image x passes through the analysis network g a is transformed to achieve the implicit representation y. The implicit representation y is quantized and compressed into bits. The number of bits R is used to measure the encoding and decoding ratio. Then the quantized implicit representation By synthesizing the network g s is inversely transformed to obtain the reconstructed image The distortion is obtained by transforming x and transforming And the function g p is calculated in the perceptual space.

[0067] It is intuitive to apply autoencoder networks to lossy compression. We just need to encode the learned latent representations from the trained neural network. However, adapting the autoencoder to images is not trivial because the original autoencoder is not optimized for compression and therefore it is not efficient by directly using the trained autoencoder. In addition, there are other major challenges: First, the low-dimensional representation should be quantized before being encoded, but quantization is not differentiable, which is required in backpropagation when training neural networks. Second, the goals in the compression scenario are different because both distortion and ratio need to be considered. Estimating the ratio is challenging. Third, practical image coding and decoding schemes need to support variable ratios, scalability, encoding speed / decoding speed, and interoperability. In response to these challenges, many researchers have actively contributed to this field.

[0068] Prototype autoencoder for image compression Figure 2 It can be regarded as a transform coding strategy. The original image x passes through the analysis network y = g a (x) is transformed, where y is the latent representation to be quantized and encoded. The synthesis network will transform the quantized latent representation Perform an inverse transform to obtain the reconstructed image The framework is trained using a rate-distortion loss function, i.e., where D is x and The distortion between them, R is expressed from the quantized is the ratio calculated or estimated, and λ is the Lagrange multiplier. It should be noted that D can be calculated in the pixel domain or the perceptual domain. All existing research works follow this prototype, and the difference may only be the network structure or the loss function.

[0069] In terms of network structure, RNN and CNN are the most widely used architectures. In the RNN-related category, a general framework for variable-rate image compression using RNN is proposed. They use binary quantization to generate codes and do not consider the ratio during training. The framework actually provides scalable codec capabilities, where RNNs with convolutional and deconvolutional layers are reported to perform appropriately. An improved version is proposed by upgrading the encoder with a neural network similar to PixelRNN to compress binary codes. It is reported that the performance is better than JPEG on the Kodak image dataset using the MS-SSIM evaluation metric. The RNN-based solution is further improved by introducing hidden state induction. In addition, an SSIM weighted loss function is also designed, and a spatially adaptive bitrate mechanism is enabled. They achieve better results than BPG on the Kodak image dataset using MS-SSIM as the evaluation metric. Spatially adaptive bitrate is supported by training a stopping-tolerant code RNN.

[0070] A general framework for rate-distortion optimized image compression is proposed. Multivariate quantization is used to generate integer codes and the ratio is considered during training, i.e., the loss is a joint rate-distortion cost, which can be MSE or other. They add random uniform noise during training to motivate quantization and use the differential entropy of the noise code as a proxy for the ratio. They use generalized dinormalization (GDN) as the network structure, which includes a linear mapping followed by a nonlinear parameter normalization. The effectiveness of GDN for image encoding and decoding is verified. An improved version is proposed, in which they use 3 convolutional layers, each followed by a downsampling layer and a GDN layer as a forward transform. Therefore, they use 3 layers of inverse GDN, each followed by an upsampling layer and a convolutional layer to motivate the inverse transform. In addition, an arithmetic coding method is designed to compress integer codes. The performance is reported to be better than JPEG and JPEG 2000 on the Kodak dataset in terms of MSE. In addition, the method is improved by designing a scale hyper-prior in the autoencoder. They use a subnet h a Transform the implicit representation y to z = h a (y) and z will be quantized and transmitted as side information. Therefore, the inverse transform is performed by trying to Decoding to quantization The standard deviation of the subnet h s is implemented, which will be in the arithmetic codec The structure in the residual space is further exploited by introducing an autoregressive model to estimate both the standard deviation and the mean. In the latest work, a Gaussian mixture model is used to further remove the redundancy in the residual. The reported performance is comparable to VVC on the Kodak image set using PSNR as the evaluation metric.

[0071] 2.3.3 Super Prior Model

[0072] In the transform codec approach to image compression, the encoder subnetwork (Section 2.3.2) uses a parametric analysis transform The image vector x is transformed into a latent representation y, which is then quantized to form because is a discrete value that can be losslessly compressed using entropy coding techniques such as arithmetic coding and transmitted as a sequence of bits.

[0073] Figure 3 An image from the Kodak dataset and different representations of the image are shown, where the left side shows an image from the Kodak dataset, the middle left shows a visualization of the latent representation y of the image, the middle right shows the standard deviation σ of the latent, and the right side shows the latent y after introducing the super-prior (super-encoder and super-decoder) network.

[0074] As from Figure 3 The middle left image and the middle right image are obvious. There is significant spatial dependence between the elements of . Notably, their scales (center right image) appear to be spatially coupled. An additional set of random variables is introduced to capture spatial dependencies and further reduce redundancy. In this case, the image compression network is Figure 4 Depicted in. Figure 4 A network architecture 400 of an autoencoder implementing a hyper-prior model is shown.

[0075] exist Figure 4 In the figure, the left hand side of the model is the encoder g a and decoder g s (Explained in Section 2.3.2). On the right hand side is the additional super encoder h a Network and for obtaining Additional super decoder h s In this architecture, the encoder inputs an image x to g a , producing a response y with a spatially varying standard deviation. The response y is fed to h a , summarizing the standard deviation distribution in z. z is then quantized The encoder then uses the quantized vector To estimate σ, the spatial distribution of the standard deviation, and use it to compress and transmit the quantized image representation The decoder first recovers the compressed signal Then it uses h s to obtain σ, which provides the correct probability estimate for the successful recovery Then it Feed to g s to obtain the reconstructed image.

[0076] When the super encoder and super decoder are added to the image compression network, the quantized latent The space redundancy is reduced. Figure 3 The rightmost image in FIG. 4 corresponds to the quantized latent when the super encoder / decoder is used. Compared to the middle right image, the spatial redundancy is significantly reduced because the samples of the quantized latent are less correlated.

[0077] Figure 4 The network architecture 400 of the autoencoder implementing the super prior model is shown. The image autoencoder network is shown on the left, and the right side corresponds to the super prior subnetwork. The analysis and synthesis transformations are denoted as g a and g a Q represents quantization, AE and AD represent arithmetic encoder and arithmetic decoder respectively. The super prior model consists of two sub-networks, the super encoder (with h a denoted) and super decoder (with h s The super prior model generates quantized super implicit quantities. It includes the quantization of implicit Information about the probability distribution of samples. is included in the bitstream and are transmitted together to the receiver (decoder).

[0078] 2.3.4 Context Model

[0079] Although the hyper-prior model improves the quantization implicit Additional improvements can be obtained by modeling the probability distribution of , by exploiting an autoregressive model that predicts the quantized latent quantity from its causal context (context model).

[0080] The term autoregressive means that the output of a process is later used as input to it. For example, the context model subnetwork generates one sample of the latent quantity, which is later used as input to get the next sample.

[0081] Joint autoregressive and hierarchical priors for learning image compression adopt a joint architecture where both the super-prior model sub-networks (super-encoder and super-decoder) and the context model sub-network are used. The super-prior and context models are combined to learn the quantized latent The probability model of is then used for entropy coding and decoding. Figure 5 As depicted in , the outputs of the context subnetwork and the super decoder subnetwork are combined through the subnetwork called entropy parameters, which generate mean μ and scale (or variance) σ parameters for the Gaussian probability model. The Gaussian probability model is then used to encode the quantized latent samples into the bitstream with the help of the Arithmetic Encoder (AE) module. In the decoder, the Gaussian probability model is utilized to obtain the quantized latent from the bitstream through the Arithmetic Decoder (AD) module

[0082] Figure 5 A block diagram 500 of a combined model is shown. The combined model jointly optimizes an autoregressive component that together with a hyper-prior and an underlying autoencoder estimates the probability distribution of the latent from its causal context (context model). The real-valued latent representation is quantized (Q) to produce a quantized latent and quantized super-implicit Quantification Implicit and quantized super-implicit Compressed into a bitstream using an Arithmetic Encoder (AE) and decompressed by an Arithmetic Decoder (AD). The highlighted areas correspond to the components executed by the receiver (ie, decoder) to recover the image from the compressed bitstream.

[0083] Typically, latent samples are modeled as Gaussian distributions or Gaussian mixture models (not limited to). In the joint autoregressive and hierarchical priors for learning image compression and according to Figure 5 , the context model and the hyper-prior are jointly used to estimate the probability distribution of latent samples. Since a Gaussian distribution can be defined by a mean and a variance (also called sigma or scale), a joint model is used to estimate the mean and variance (denoted as μ and σ).

[0084] 2.3.5 Gain Variational Autoencoder (G-VAE)

[0085] Typically, neural network-based image / video compression methods require training multiple models to adapt to different rates. Gain Variational Autoencoder (G-VAE) is a variational autoencoder with a pair of gain units, which is designed to achieve continuous variable rate adaptation using a single model. It includes a pair of gain units that are usually inserted at the output of the encoder and the input of the decoder. The output of the encoder is defined as the latent representation y∈R c*h*w , where c, h, w represent the number of channels, the height and width of the latent representation. Each channel of the latent representation is represented as y(i) ∈R h*w , where i = 0, 1, ..., c-1. A pair of gain units includes a gain matrix R∈R c*n and the inverse gain matrix, where n is the number of gain matrices. The gain vector can be represented as m s ={α s(0) ,α s(1) ,...,α s(c-1)},α s(i) ∈R, where S represents the index of the gain vector in the gain matrix.

[0086] The motivation of the gain matrix is ​​similar to the quantization table in JPEG, which controls the quantization loss based on the characteristics of different channels. To apply the gain matrix to the latent representation, each channel is multiplied by the corresponding value in the gain vector.

[0087]

[0088] where ⊙ is the channel-by-channel multiplication, i.e. And α s(i) is the gain vector m s The inverse gain matrix used at the decoder side can be expressed as M′∈R c*n , which consists of n inverse gain vectors, namely, M′={δ s(0) ,δ s(1) ,...,δ s(c-1)}, δ s(i) ∈R. The inverse gain process is expressed as

[0089]

[0090] in is the decoded quantized implicit representation, and y′ s is the quantized implicit representation of the inverse gain that will be fed into the synthesis network.

[0091] To achieve continuously variable rate adjustment, interpolation is used between the vectors. Given two pairs of gain vectors {m t , m′ t} and {m r , m′ r}, the interpolated gain vector can be obtained via the following equation.

[0092] m v =[(m r ) l ·(m t ) 1-l ]

[0093] m′ v =[(m′ r ) l·(m′ t ) 1-l ]

[0094] where l∈R is the interpolation coefficient, which controls the corresponding bit rate of the generated gain vector pair. Since l is a real number, any bit rate between a given pair of gain vectors can be achieved.

[0095] 2.3.6 Encoding process using joint autoregressive hyper-prior model

[0096] Figure 5 Corresponding to the state of the art compression methods proposed, this is proposed in Joint Autoregressive and Hierarchical Priors for Learning Image Compression. In this section and the next, the encoding process and the decoding process will be described separately.

[0097] Figure 6 Depicted is an encoding process 600 based on joint autoregressive and hierarchical priors for learning image compression. An input image is first processed through an encoder subnetwork. The encoder transforms the input image into a transformed representation called a latent, denoted by y. y is then input to a quantizer block denoted by Q to obtain a quantized latent (y^). y^ is then converted into a bitstream (bitstream1) using an arithmetic coding module (denoted as AE). The arithmetic coding block converts each sample of y^ into a bitstream (bitstream1) one by one in a sequential order.

[0098] The modules super encoder, context, super decoder and entropy parameter sub-network are used to estimate the probability distribution of samples of the quantized hidden quantity y^. The hidden quantity y is input to the super encoder, which outputs the super hidden quantity (denoted by z). The super hidden quantity is then quantized ((z^)), and a second bit stream (bit stream 2) is generated using the arithmetic coding (AE) module. The decomposition entropy module generates a probability distribution, which is used to encode the quantized super hidden quantity into the bit stream. The quantized super hidden quantity includes information about the probability distribution of the quantized hidden quantity (y^).

[0099] The entropy parameter subnetwork generates a probability distribution estimate that is used to encode the quantized latent quantity y^. The information generated by the entropy parameter typically includes the mean μ and scale (or variance) σ parameters, which are used together to obtain a Gaussian probability distribution. The Gaussian distribution of a random variable x is defined as where the parameter μ is the mean or expectation of the distribution (as well as its median and mode), while the parameter σ is its standard deviation (or variance or scale). To define a Gaussian distribution, the mean and variance need to be determined. In the joint autoregressive and hierarchical prior for learning image compression, the entropy parameter module is used to estimate the mean and variance values.

[0100] The subnetwork superdecoder generates part of the information used by the entropy parameter subnetwork, and the other part of the information is generated by an autoregressive module called a context module. The context module uses samples that have been encoded by the arithmetic coding (AE) module to generate information about the probability distribution of samples of the quantized hidden quantity. The quantized hidden quantity y^ is usually a matrix consisting of many samples. The sample can be indicated using an index, such as y^[i, j, k] or y^[i, j], depending on the dimension of the matrix y^. The samples y^[i, j] are encoded one by one by the AE, usually using a raster scan order. In the raster scan order, the rows of the matrix are processed from top to bottom, where the samples in a row are processed from left to right. In this scenario (where the raster scan order is used by the AE to encode samples into the bitstream), the context module uses the samples that are encoded before the raster scan order to generate information about the sample y^[i, j]. The information generated by the context module and the superdecoder is combined by the entropy parameter module to generate a probability distribution for encoding the quantized hidden quantity y^ into the bitstream (bitstream 1).

[0101] Finally, as a result of the encoding process, the first bitstream and the second bitstream are transmitted to a decoder.

[0102] Note that other names can be used for the above modules.

[0103] In the above description, Figure 6 All elements in are collectively called encoders. The analysis transformation that transforms the input image into a latent representation is also called an encoder (or autoencoder).

[0104] 2.3.7 Decoding process using the joint autoregressive hyper-prior model

[0105] Figure 7 Depicted are decoding processes 700 corresponding to joint autoregressive and hierarchical priors for learning image compression, respectively.

[0106] During the decoding process, the decoder first receives a first bit stream (bit stream 1) and a second bit stream (bit stream 2) generated by the corresponding encoder. Bit stream 2 is first decoded by an arithmetic decoding (AD) module by using a probability distribution generated by a decomposition entropy subnetwork. The decomposition entropy module typically generates a probability distribution using a predetermined template, such as a predetermined mean and variance value in the case of a Gaussian distribution. The output of the arithmetic decoding process of bit stream 2 is z^, which is a quantized super-hidden amount. The AD process is restored to the AE process applied in the encoder. The AE and AD processes are lossless, which means that the quantized super-hidden amount z^ generated by the encoder can be reconstructed at the decoder without any changes.

[0107] After z^ is obtained, it is processed by the super decoder, and the output of the super decoder is fed to the entropy parameter module. The three sub-networks used in the decoder, the context, super decoder and entropy parameters are the same as the context, super decoder and entropy parameter sub-networks in the encoder. Therefore, exactly the same probability distribution can be obtained in the decoder (as in the encoder), which is necessary for reconstructing the quantized hidden y^ without any loss. As a result, the same version of the quantized hidden y^ obtained in the decoder can be obtained in the encoder.

[0108] Following the probability distribution (e.g., mean and variance parameters) obtained by the entropy parameter subnetwork, the arithmetic decoding module decodes the quantized latent samples one by one from the bitstream of bitstream 1. From a practical point of view, the autoregressive model (context model) is inherently serial and therefore cannot be accelerated using techniques such as parallelization.

[0109] Finally, the fully reconstructed quantized latent y^ is input to the synthesis transform (in Figure 7 The decoder module is used to obtain the reconstructed image.

[0110] In the above description, Figure 7 All elements in are collectively referred to as decoders. The synthetic transform that transforms the quantized latent into the reconstructed image is also called a decoder (or autodecoder).

[0111] 2.4 Neural Networks for Video Compression

[0112] Similar to conventional video codec techniques, neural images serve as the basis for intra-frame compression in neural network-based video compression, so the development of neural network-based video compression technology is later than that of neural network-based image compression, but more efforts are needed to solve the challenges due to its complexity. Since 2017, several researchers have been working on neural network-based video compression schemes. Compared with image compression, video compression requires effective methods to remove inter-picture redundancy. Inter-picture prediction is then a key step in these works. Motion estimation and compensation are widely adopted, but have only recently been implemented by trained neural networks.

[0113] The research on neural network-based video compression can be divided into two categories according to the target scenario: random access and low latency. In the case of random access, the decoding can be started from any point in the sequence, usually the whole sequence is divided into multiple separate segments, and each segment can be decoded independently. In the case of low latency, it aims to reduce the decoding time, so that usually only the previous frame in time can be used as a reference frame to decode the subsequent frame.

[0114] 2.4.1 Low Latency

[0115] A video compression scheme with trained neural network is proposed. They first divide the video sequence frames into blocks, and each block will choose one of the two available modes, intra codec or inter codec. If intra codec is selected, there is an associated autoencoder to compress the block. If inter codec is selected, motion estimation and compensation are performed by traditional methods, and the trained neural network will be used for residual compression. The output of the autoencoder is directly quantized and encoded by the Huffman method.

[0116] Another neural network based video codec scheme with PixelMotionCNN is proposed. Frames are compressed in temporal order and each frame is divided into blocks which are compressed in raster scan order. Each frame will first be extrapolated by the two previous reconstructed frames. When a block is to be compressed, the extrapolated frame along with the context of the current block is fed into PixelMotionCNN to derive the latent representation. The residual is then compressed by a variable ratio image scheme. The scheme is comparable to that performed on H.264.

[0117] A video compression framework based on a truly end-to-end neural network (e.g., an end-to-end deep video compression framework DVC) is proposed, in which all modules are implemented through neural networks. The scheme accepts the current frame and the previously reconstructed frame as input, and the optical flow will be derived as motion information through a pre-trained neural network. The motion information will be warped together with the reference frame, followed by a neural network that generates a motion compensated frame. The residual and motion information are compressed through two separate neural autoencoders. The entire framework is trained with a single rate-distortion loss function. It achieves better performance than H.264.

[0118] An advanced neural network based video compression scheme is proposed based on existing designs. It inherits and extends the traditional video codec scheme of neural networks with the following main features: 1) only one autoencoder is used to compress motion information and residuals; 2) motion compensation with multiple frames and multiple optical flows; 3) online states are learned and propagated through the following frames over time. This scheme achieves better performance than the HEVC reference software in MS-SSIM.

[0119] Based on the end-to-end deep video compression framework DVC, an extended end-to-end neural network-based video compression framework is proposed. In this solution, multiple frames are used as references. Therefore, it is possible to provide a more accurate prediction of the current frame by using multiple reference frames and associated motion information. In addition, motion field prediction is deployed to remove motion redundancy along the temporal channel. The post-processing network is also introduced in this work to remove reconstruction artifacts from the previous process. In terms of both PSNR and MS-SSIM, the performance is better than DVC and H.265.

[0120] A scale-space stream is proposed to replace the commonly used optical flow by adding a scale parameter through a DVC-based framework. It is reported to achieve better performance than H.264.

[0121] A multi-resolution representation for optical flow is proposed based on DVC. Specifically, the motion estimation network generates multiple optical flows with different resolutions and lets the network learn which one to choose under the loss function. The performance is slightly improved relative to DVC and better than H.265.

[0122] 2.4.2 Random Access

[0123] A neural network-based video compression scheme with frame interpolation is proposed. The key frame is first compressed by a neural image compressor and the remaining frames are compressed in a hierarchical order. They perform motion compensation in the perceptual domain, i.e., feature maps are derived at multiple spatial scales of the original frame and the motion is used to warp the feature maps, which are then fed into the image compressor. The method is reported to be comparable to H.264.

[0124] An interpolation-based video compression method is proposed, where the interpolation model combines motion information compression and image synthesis, and the same autoencoder is used for both the image and the residual.

[0125] A neural network-based video compression method based on variational autoencoders and deterministic encoders is proposed. Specifically, the model consists of an autoencoder and an autoregressive prior. Unlike previous methods, this method accepts a group of pictures (GOP) as input and incorporates a 3D autoregressive prior by considering temporal correlation while encoding and decoding implicit representations. It provides performance comparable to H.265.

[0126] 2.5. Preliminary Concept

[0127] Almost all natural images / videos are in digital format. Grayscale digital images can be obtained by To indicate that is a collection of pixel values, m is the image height, and n is the image width. For example, is a common setting, in which case Therefore, pixels can be represented by 8-bit integers. Uncompressed grayscale digital images have 8 bits per pixel (bpp), and compressed ones will certainly have fewer bits.

[0128] Color images are usually represented by multiple channels to record color information. For example, in the RGB color space, an image can be represented as Three separate channels store red, green, and blue information. Similar to an 8-bit grayscale image, an uncompressed 8-bit RGB image has 24bpp. Digital images / videos can be represented in different color spaces. Most neural network-based video compression schemes are developed in RGB color space, while traditional codecs typically use YUV color space to represent video sequences. In the YUV color space, the image is decomposed into three channels, namely Y, Cb, and Cr, where Y is the brightness component and Cb / Cr are the chrominance components. The advantage is that Cb and Cr are usually downsampled for pre-compression because the human visual system is less sensitive to the chrominance components.

[0129] A color video sequence consists of multiple color images (called frames) that record scenes at different time stamps. For example, in the RGB color space, a color video can be represented as: X = {x 0 , x 1 , ..., x t , ..., x T-1}, where T is the number of frames in the video sequence, If m=1080, n=1920, And the video has 50 frames per second (fps), then the data rate of the uncompressed video is: 1920×1080×8×3×50=2,488,320,000 bits per second (bps), which is about 2.32Gpbs. This requires a large amount of storage, so it must be compressed before transmission on the Internet.

[0130] Typically, for natural images, lossless methods can achieve a compression ratio of about 1.5 to 3, which is clearly lower than required. Therefore, in order to achieve further compression ratios, lossy compression was developed, but at the cost of introducing distortion. Distortion can be measured by calculating the average squared difference between the original image and the reconstructed image, i.e., the mean square error (MSE). For grayscale images, MSE can be calculated using the following formula.

[0131]

[0132] Therefore, the quality of the reconstructed image compared to the original image can be measured by the Peak Signal-to-Noise Ratio (PSNR):

[0133]

[0134] in yes The maximum value in , for example 255 for an 8-bit grayscale image. There are other quality assessment metrics such as structural similarity (SSIM) and multi-scale SSIM (MS-SSIM).

[0135] To compare different lossless compression schemes, it is sufficient to compare the compression ratio given the resulting rate, and vice versa. However, to compare different lossy compression methods, both the rate and the reconstruction quality must be considered. For example, calculating the relative rates at several different quality levels and then averaging these rates is a common approach; this average relative rate is also called Bjontegaard's delta-rate (BD-rate). There are other important aspects of evaluating video / image coding schemes, including encoding / decoding complexity, scalability, robustness, etc.

[0136] 3. Question

[0137] 3.1 Core Issues

[0138] Problem 1. The common way to achieve different rates in neural network-based image / video compression is to train multiple models with separate Lagrange multipliers, which is neither time-efficient nor storage-efficient. In addition, using multiple fixed-rate pre-trained models can only achieve discrete rate points.

[0139] Problem 2. Prior art works typically process multiple components (i.e., RGB or YUV) in a single network, which may hurt performance because these components exhibit different characteristics.

[0140] 4. Detailed solution

[0141] The detailed embodiments below should be considered as examples to explain the overall concept. These embodiments should not be interpreted in a narrow manner. In addition, these embodiments can be combined in any way. Separate networks are used to process luma and chroma components. The benefits are twofold: 1) Luma and chroma have different characteristics, so that using separate networks can better capture their unique patterns and improve codec performance. 2) The chroma component contains less information than the luma component, so using a simpler network can reduce decoding complexity.

[0142] 4.1 Objectives of this Disclosure

[0143] The goal of the present disclosure is to provide a continuously variable rate neural network based image / video compression method using one or several models.

[0144] 4.2 Core of this disclosure

[0145] 4.2.1 Decoding process

[0146] Figure 8 A decoding process 800 is shown according to an embodiment of the present disclosure.

[0147] According to an embodiment, the decoding operation is performed as follows:

[0148] 1. First, a factorized entropy model is used to decode the quantized latent quantities for luminance and chrominance, i.e., Figure 8 In and

[0149] 2. Generate a quantized residual implicit by performing an arithmetic decoding process using the probability parameters (eg, variance) generated by the second network.

[0150] 3. Such as Figure 8 As shown in FIG. 1 , the inverse gain unit (iGain) is used to inversely gain the quantized residual implicit. The output of the inverse gain unit is represented as and

[0151] 4. For the brightness component, perform the following steps in a loop until you get So far as all elements of:

[0152] a. Use the first subnetwork by using The samples already obtained are used to estimate the quantized latent The mean parameter of .

[0153] b. Using quantized residual implicit and the mean to obtain The next element of .

[0154] 5. After obtaining After all samples of , the synthesis transform can be applied to obtain the reconstructed image.

[0155] 6. For the chroma components, steps 4 and 5 are the same, but utilize a different set of networks.

[0156] 7. Use the decoded luminance component as additional information to obtain the chrominance component. Specifically, the inter-channel-related information filtering sub-network (ICCI) is used for chrominance component recovery. Luminance is fed into the ICCI sub-network as additional information to assist in chrominance component decoding.

[0157] 8. After reconstructing the luma and chroma components, perform Adaptive Color Transform (ACT).

[0158] The module named ICCI is a post-processing module based on a neural network. The embodiment is not limited to the ICCI sub-network, and any other post-processing module based on a neural network may also be used.

[0159] On the top Figure 8An exemplary implementation of the embodiment is depicted in (Decoding Process). The framework includes two branches for luma component and chroma component respectively. In each branch, the first subnetwork includes context, prediction and optional super decoder modules. The second network includes a super scale decoder module. The quantized super implicit is and The arithmetic decoding process generates a quantized residual implicit, which is further fed into the iGain unit to obtain the gained quantized residual implicit and

[0160] After obtaining the residual latent, a recursive prediction operation is performed to obtain the latent and The following steps describe how to obtain the implicit and processes the chrominance components in the same way but using different networks.

[0161] 1. Use the autoregressive context module by using samples Generate the first input of the prediction module, where the (m,n) pair is the index of the latent sample that has been obtained.

[0162] 2. Optionally, by using a super decoder and quantization super concealment To obtain the second input of the prediction module.

[0163] 3. Using the first input and the second input, the prediction module generates the mean mean[:,i,j].

[0164] 4. The mean mean[:,i,j] and the quantized residual implicit Add together to get the implicit

[0165] 5. Repeat steps 1-4 for the next sample.

[0166] Whether and / or how to apply at least one method disclosed in this document may be signaled from an encoder to a decoder, for example in a bitstream.

[0167] Alternatively, whether and / or how to apply at least one method disclosed in the document may be determined by a decoder based on codec information such as dimension, color format, etc.

[0168] Alternatively or additionally, the designation MS1, MS2 or MS3+O (in Figure 8 A module (in) may be included in the process flow. The module may perform an operation on its input to obtain an output by multiplying the input with a scalar or adding an additive component to the input. The scalar or additive component used by the module may be indicated in the bitstream.

[0169] Figure 8 The module named RD or the module named AD in the example may be an entropy decoding module. It may be a range decoder or an arithmetic decoder, etc.

[0170] The embodiments described herein are not limited to Figure 8 Some modules may be missing, and some modules may be shifted in the processing order. Additional modules may also be included. For example:

[0171] 1. The ICCI module can be removed. In this case, the outputs of the Synthesize module and the Synthesize UV ​​module can be combined by another module, which can be based on a neural network.

[0172] 2. One or more of the modules named MS1, MS2 or MS3+0 may be removed. The core of the present disclosure is not affected by the removal of one or more of the scaling and addition modules.

[0173] 4.3 Differences between the State of the Art and the Present Disclosure

[0174] There are three main differences between the state of the art and embodiments according to the present disclosure.

[0175] 1. The embodiment includes both gain units and decoupled architectures, ie, arithmetic encoding and decoding are performed independently of the autoregressive sub-network. Thus, the embodiment can provide better parallelization with continuous rate adaptation.

[0176] 2. Use two separate networks to process the luminance component and the chrominance component.

[0177] 3. Chroma resampling with the luma component using a post-processing filter (eg ICCI).

[0178] 4.4 Benefits of the Implementation

[0179] The beneficial effects of the embodiment are as follows:

[0180] 1. A single pre-trained model can be used to achieve continuous rate.

[0181] 2. Dedicated networks for luma and chroma components can capture their unique characteristics more efficiently, thereby improving compression performance.

[0182] 4.5 Examples

[0183] The following detailed embodiments should be considered as examples to explain the general concept. These embodiments should not be interpreted in a narrow manner. In addition, these embodiments can be combined in any way.

[0184] 1. In one example, the decoder includes two branches, one for the luma component and the other for the chroma component. The luma component is used to help the reconstruction of the chroma component.

[0185] 2. In one example, the decoder includes both a decoupled architecture and a gain unit for better penalty and continuous rate adaptation, respectively.

[0186] 3. In one example, the chroma components are resampled using an ICCI filter.

[0187] 5. Examples

[0188] 1.Decoder:

[0189] A method for decoding an image or video comprises the following steps:

[0190] - Use the bitstream and the output of the first subnetwork with the inverse gain unit (iGain) to obtain the quantization

[0191] Samples of residual hidden quantities The first sub-network is not autoregressive.

[0192] -Use the second subnetwork and quantized implicit The reconstructed samples obtain the predicted value mean.

[0193] -Use quantized residual samples And the predicted mean reconstructs the quantized latent The next sample of .

[0194] Using quantized implicit And synthesize the transformation to obtain the reconstructed image.

[0195] As mentioned above, the common way to achieve different rates in neural network-based image / video compression is to train multiple models with separate Lagrange multipliers, which is neither time-efficient nor memory-efficient. In addition, using multiple fixed-rate pre-trained models can only achieve discrete rate points.

[0196] In order to solve the above problems and some unmentioned problems, a visual data processing solution as described below is disclosed. The embodiments of the present disclosure should be considered as examples to explain the overall concept and should not be interpreted in a narrow manner. In addition, these embodiments can be applied individually or in combination in any way.

[0197] Fig. 9 A flow chart of a method 900 for visual data processing according to an embodiment of the present disclosure is shown. The method 900 is implemented for conversion between visual data and at least one bit stream of visual data.

[0198] At block 910, a residual representation of the visual data is determined based on at least a gain parameter and a first probability distribution parameter of the visual data, such as a variance of the visual data. The residual representation represents a residual value compared to a second probability distribution representation of the visual data, such as a mean of the visual data. The gain parameter adjusts a value range of the residual representation.

[0199] At block 920, a conversion is performed based on the residual representation. In some embodiments, the conversion includes encoding the visual data into at least one bitstream. Alternatively or additionally, in some embodiments, the conversion includes decoding the visual data from at least one bitstream.

[0200] Method 900 implements a continuously variable rate of the output of model-based visual data processing. For example, the rate of at least one bit stream or the rate of visual data can be adjusted to a continuously variable rate. Therefore, the coding efficiency and coding effectiveness can be improved.

[0201] In some embodiments, at least one bitstream includes a first bitstream, such as bitstream 1 (also referred to as bits1) and a second bitstream (also referred to as bits2). The first bitstream includes arithmetic information of the visual data, and the second bitstream includes probability distribution information of the visual data.

[0202] In some embodiments, the conversion includes decoding the visual data from the first bit stream and the second bit stream. In the decoding conversion, the gain parameter includes a first gain parameter. The first probability distribution parameter of the visual data can be determined based on the second bit stream and a model such as a superscale decoder. The residual representation of the visual data can be determined based on the first bit stream, the first probability distribution parameter, and the first gain parameter. The visual data can be determined based on at least the residual representation of the visual data. For example, the probability parameter (e.g., variance) generated by the second network is used to generate a quantized residual implicit by performing an arithmetic decoding process. Figure 8 As shown in FIG. 1 , the inverse gain unit (iGain) is used to perform inverse gain quantization on the quantized residual implicit. The output of the inverse gain unit is represented as and In obtaining After all samples of , the synthesis transform can be applied to obtain the reconstructed image.

[0203] In some embodiments, obtaining the residual representation of the visual data comprises: determining a first intermediate representation of the residual representation based on the first bitstream, the first probability distribution parameters and the first entropy model; and determining the residual representation by multiplying the first gain parameter by the first intermediate representation.

[0204] In some embodiments, the first intermediate representation includes multiple components associated with multiple channels of the first entropy model. The first gain parameter includes multiple gain components. The number of the multiple gain components is the same as the number of the multiple channels. The first entropy model can be a range decoder or an arithmetic decoder.

[0205] In some embodiments, determining the visual data includes: determining a second probability distribution representation of the visual data based on the prediction module and the first sample of the visual data; determining a second sample of the visual data based on the residual representation and the second probability distribution representation of the visual data; and reconstructing the visual data based on the first sample and the second sample and the synthesis model. For example, the first subnetwork is used to use The samples already obtained are used to estimate the quantized latent The mean parameter of . and the mean is used to get the next element In obtaining After all samples of , the synthesis transform can be applied to obtain the reconstructed image.

[0206] In some embodiments, determining the second probability distribution representation includes: determining context information of the visual data based on the first sample and the context module; determining super information of the visual data based on at least the second bitstream and the second entropy model; and determining the second probability distribution representation based on the context information, the super information and the prediction module. In some embodiments, the super information is further determined based on the super encoder. The second entropy model can be a range decoder or an arithmetic decoder.

[0207] In the example, the autoregressive context module is used to use the sample The first input to the prediction module is generated where the (m,n) pair is the index of the already obtained sample of the hidden quantity. Optionally, the second input to the prediction module is obtained by using the super decoder and quantizing the super hidden quantity. Using the first and second inputs, the prediction module generates the mean mean[:,i,j].

[0208] In some embodiments, determining the first probability distribution parameter of the visual data includes: determining a variance value of the visual data based on the second bitstream and the model; and determining the first probability distribution parameter based on the variance value and a first predefined value, the first predefined value being added to or multiplied by the variance value. For example, the first predetermined value may be determined by Figure 8 The module MS1 shown is implemented.

[0209] In some embodiments, obtaining the residual representation includes: determining a second intermediate representation of the visual data based on the first bitstream, the first probability distribution parameter, and the first gain parameter; and determining the residual representation based on the second intermediate representation and a second predefined value, the second predefined value being added to or multiplied by the second intermediate representation. For example, the second predetermined value may be determined by Figure 8 The module MS2 shown is implemented.

[0210] In some embodiments, the model includes a superscale encoder that is not autoregressive.

[0211] In some embodiments, the residual representation of the visual data includes a luma residual representation and a chroma residual representation.

[0212] In some embodiments, the visual data includes a luma component and a chroma component. The visual data may be determined by determining the luma component based at least on a luma residual representation; and determining the chroma component based at least on a chroma residual representation. That is, the decoder may include two branches, one for the luma component and the other for the chroma component.

[0213] For example, Figure 8 The framework shown includes two branches for luma and chroma components respectively. In each branch, the first subnetwork includes context, prediction and optional super decoder modules. The second network includes super scale decoder module. The quantized super implicit is and The arithmetic decoding process generates quantized residual implicits, which are further fed into the iGain unit to obtain the gained quantized residual implicits and After obtaining the residual latent, a recursive prediction operation is performed to obtain the latent and

[0214] In some embodiments, determining the chrominance component includes: determining the chrominance component based on the chrominance residual representation and the luma component. That is, the luma component can be used to help reconstruct the chrominance component. The decoded luma component can be used as additional information to obtain the chrominance component.

[0215] In some embodiments, a neural network based post-processing module is applied to the luma component, and the chroma component is determined based on the post-processed luma component. In some embodiments, the neural network based post-processing module includes an inter-channel correlation information (ICCI) codec. For example, the chroma component can be resampled using an ICCI filter. Specifically, ICCI can be used for chroma component recovery. Luma is fed into the ICCI subnetwork as additional information to assist in chroma component decoding.

[0216] In some embodiments, the luma residual representation is determined based on a first luma probability distribution parameter and a first gain parameter.

[0217] In some embodiments, the first luminance probability distribution parameters are determined based on a first superscale codec.

[0218] In some embodiments, the luma component is determined based on a luma residual representation, a first context model, and a first prediction module. In some embodiments, the luma component is further based on a first supercodec.

[0219] In some embodiments, the chroma residual representation is determined based on a first chroma probability distribution parameter and a first gain parameter.

[0220] In some embodiments, the first chroma probability distribution parameters are determined based on a second superscale codec.

[0221] In some embodiments, the chroma components are determined based on a chroma residual representation, a second context model, and a second prediction module. In some embodiments, the chroma components are further based on a second supercodec.

[0222] In some embodiments, the method 900 further includes: applying an adaptive color transform to the luma component and the chroma component; and determining visual data based on the application. That is, the adaptive color transform (ACT) is performed after reconstructing the luma component and the chroma component.

[0223] In some embodiments, the method 900 further includes: updating at least one of the luminance component or the chrominance component based on a third predefined value, the third predefined value being added to or multiplied by at least one of the luminance component or the chrominance component. For example, the third predefined value may be Figure 8 The module shown is implemented in MS3+O.

[0224] In some embodiments, the conversion includes encoding the visual data into a first bitstream and a second bitstream, and the gain parameter includes a second gain parameter that is opposite to the first gain parameter (associated with the decoding conversion). In the encoding conversion, the first bitstream and the second bitstream can be determined based on at least the second gain parameter and the residual representation.

[0225] The example embodiment implements a gain unit and a decoupled architecture, i.e., performs arithmetic coding and decoding independently of the autoregressive sub-network. Therefore, it can provide better parallelization with continuous rate adaptation.

[0226] In some embodiments, the residual representation comprises a quantized residual representation.

[0227] In some embodiments, the gain parameter includes a plurality of gain vectors. For example, the gain parameter may be a gain matrix or a gain unit. The gain vector in the plurality of gain vectors has a plurality of gain components. The visual data includes a plurality of components associated with a plurality of channels. The number of the plurality of gain components may be the same as the number of the plurality of channels.

[0228] In some embodiments, the third gain vector is determined based on a convolution of a first gain vector and a second gain vector of the plurality of gain vectors.

[0229] In some embodiments, the first information about applying the method is included in the bitstream.For example, whether and / or how to apply at least one method disclosed in this document can be signaled from an encoder to a decoder, for example in the bitstream.

[0230] In some embodiments, method 900 further includes: determining first information about applying the method based on the codec information. In some embodiments, the codec information includes at least one of the dimension of the visual data or the color format of the visual data. For example, whether and / or how to apply at least one method disclosed in the document can be determined by the decoder based on the codec information such as the dimension, the color format, etc.

[0231] In some embodiments, the first information indicates at least one of: how to apply the method, or whether to apply the method.

[0232] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of visual data generated by a method performed by a visual data processing device. According to the method, a residual representation of the visual data is determined based on at least a gain parameter and a first probability distribution parameter of the visual data. The residual representation represents a residual value compared to a second probability distribution representation of the visual data. The gain parameter adjusts a value range of the residual representation. At least one bitstream is generated based on the residual representation.

[0233] According to some further embodiments of the present disclosure, a method for storing a bitstream of visual data is provided. In the method, a residual representation of the visual data is determined based at least on a gain parameter and a first probability distribution parameter of the visual data. The residual representation represents a residual value compared to a second probability distribution representation of the visual data. The gain parameter adjusts a value range of the residual representation. At least one bitstream is generated based on the residual representation. The at least one bitstream is stored in a non-transitory computer-readable recording medium.

[0234] Implementations of the present disclosure may be described according to the following clauses, and features thereof may be combined in any reasonable manner. By using implementations or embodiments of the present disclosure, a single pre-trained model may be utilized to achieve continuous rate. In addition, dedicated networks for luma and chroma components may more efficiently capture their unique characteristics, thereby improving compression performance.

[0235] Clause 1. A method for processing visual data, comprising: for conversion between at least one bit stream of visual data and the visual data, determining a residual representation of the visual data based at least on a gain parameter and a first probability distribution parameter of the visual data, the residual representation representing a residual value compared to a second probability distribution representation of the visual data, the gain parameter adjusting a value range of the residual representation; and performing the conversion based on the residual representation.

[0236] Clause 2. The method of clause 1, wherein the at least one bitstream comprises a first bitstream and a second bitstream, the first bitstream comprising arithmetic information of the visual data, the second bitstream comprising probability distribution information of the visual data.

[0237] Clause 3. A method according to Clause 2, wherein the conversion includes decoding the visual data from the first bit stream and the second bit stream, the gain parameters include a first gain parameter, wherein determining the residual representation includes: determining the first probability distribution parameter of the visual data based on the second bit stream and a model; and obtaining the residual representation of the visual data based on the first bit stream, the first probability distribution parameter and the first gain parameter; wherein performing the conversion includes: determining the visual data based at least on the residual representation of the visual data.

[0238] Clause 4. A method according to Clause 3, wherein obtaining the residual representation of the visual data comprises: determining a first intermediate representation of the residual representation based on the first bitstream, the first probability distribution parameters and a first entropy model; and determining the residual representation by multiplying the first gain parameter by the first intermediate representation.

[0239] Clause 5. A method according to clause 4, wherein the first intermediate representation comprises multiple components associated with multiple channels of the first entropy model, and the first gain parameter comprises multiple gain components, the number of the multiple gain components is the same as the number of the multiple channels.

[0240] Clause 6. A method according to any one of clauses 3-5, wherein determining the visual data comprises: determining the second probability distribution representation of the visual data based on a prediction module and a first sample of the visual data; determining a second sample of the visual data based on the residual representation and the second probability distribution representation of the visual data; and reconstructing the visual data based on the first sample, the second sample and a synthesis model.

[0241] Clause 7. A method according to Clause 6, wherein determining the second probability distribution representation comprises: determining context information of the visual data based on the first sample and the context module; determining super information of the visual data based at least on the second bitstream and the second entropy model; and determining the second probability distribution representation based on the context information, the super information and the prediction module.

[0242] Clause 8. The method of clause 7, wherein the super information is further determined based on a super codec.

[0243] Clause 9. A method according to any one of clauses 3-8, wherein determining the first probability distribution parameter of the visual data comprises: determining a variance value of the visual data based on the second bit stream and the model; and determining the first probability distribution parameter based on the variance value and a first predefined value, the first predefined value being added to the variance value or multiplied by the variance value.

[0244] Clause 10. A method according to any one of clauses 3-9, wherein obtaining the residual representation comprises: determining a second intermediate representation of the visual data based on the first bitstream, the first probability distribution parameter and the first gain parameter; and determining the residual representation based on the second intermediate representation and a second predefined value, the second predefined value being added to the second intermediate representation or multiplied by the second intermediate representation.

[0245] Clause 11. A method according to any of clauses 3-10, wherein the model includes a superscale codec, the superscale codec is not autoregressive.

[0246] Clause 12. A method according to any of clauses 3-11, wherein the residual representation of the visual data comprises a luma residual representation and a chroma residual representation.

[0247] Clause 13. A method according to clause 12, wherein the visual data includes a luminance component and a chrominance component, and wherein determining the visual data based at least on the residual representation of the visual data includes: determining the luminance component based at least on the luminance residual representation; and determining the chrominance component based at least on the chrominance residual representation.

[0248] Clause 14. The method of clause 13, wherein determining the chroma component comprises determining the chroma component based on the chroma residual representation and the luma component.

[0249] Clause 15. The method of clause 14, wherein a neural network-based post-processing module is applied to the luma component, and the chroma components are determined based on the post-processed luma component.

[0250] Clause 16. The method of clause 15, wherein the neural network-based post-processing module comprises an inter-channel correlation information (ICCI) codec, and the chroma components are resampled by the ICCI codec.

[0251] Clause 17. The method of any of clauses 12-16, wherein the luma residual representation is determined based on a first luma probability distribution parameter and the first gain parameter.

[0252] Clause 18. The method of clause 17, wherein the first luminance probability distribution parameter is determined based on a first superscale codec.

[0253] Clause 19. A method according to any of clauses 13-18, wherein the luma component is determined based on the luma residual representation, a first context model and a first prediction module.

[0254] Clause 20. The method of clause 19, wherein the luma component is further based on a first supercodec.

[0255] Clause 21. A method according to any of clauses 12-20, wherein the chroma residual representation is determined based on a first chroma probability distribution parameter and the first gain parameter.

[0256] Clause 22. The method of clause 21, wherein the first chroma probability distribution parameters are determined based on a second superscale codec.

[0257] Clause 23. A method according to any of clauses 13-22, wherein the chroma component is determined based on the chroma residual representation, a second context model and a second prediction module.

[0258] Clause 24. The method of clause 23, wherein the chroma component is also based on a second supercodec.

[0259] Clause 25. The method of any of clauses 13-24, further comprising: applying an adaptive color transform to the luma component and chroma components; and determining the visual data based on the applying.

[0260] Clause 26. The method according to any one of clauses 13-25 further includes: updating at least one of the luminance component and the chrominance component based on a third predefined value, wherein the third predefined value is added to the luminance component or the at least one of the chrominance components or multiplied by the luminance component or the at least one of the chrominance components.

[0261] Clause 27. A method according to clause 2, wherein the conversion includes encoding the visual data into the first bitstream and the second bitstream, the gain parameter includes a second gain parameter that is opposite to the first gain parameter, the first gain parameter is associated with a decoding conversion, and wherein performing the conversion includes: determining the first bitstream and the second bitstream based on at least the second gain parameter and the residual representation.

[0262] Clause 28. The method of any of clauses 1-26, wherein the residual representation comprises a quantized residual representation.

[0263] Clause 29. A method according to any one of clauses 1-28, wherein the gain parameters include multiple gain vectors, a gain vector in the multiple gain vectors has multiple gain components, the visual data includes multiple components associated with multiple channels, and the number of the multiple gain components is the same as the number of the multiple channels.

[0264] Clause 30. The method of clause 29, wherein a third gain vector is determined based on a convolution of a first gain vector and a second gain vector of the plurality of gain vectors.

[0265] Clause 31. The method according to any of clauses 1-30, wherein first information about applying the method is included in the bitstream.

[0266] Clause 32. The method according to any one of clauses 1-30, further comprising: determining first information about applying the method based on codec information.

[0267] Clause 33. The method of clause 32, wherein the codec information comprises at least one of: a dimension of the visual data, or a color format of the visual data.

[0268] Clause 34. A method according to any one of clauses 31-33, wherein the first information indicates at least one of: how to apply the method, or whether to apply the method.

[0269] Clause 35. The method of any one of clauses 1-34, wherein the converting comprises encoding the visual data into the at least one bitstream.

[0270] Clause 36. The method of any of clauses 1-34, wherein the converting comprises decoding the visual data from the at least one bitstream.

[0271] Clause 37. A visual data processing device comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of clauses 1-36.

[0272] Clause 38. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method of any of clauses 1-36.

[0273] Item 39. A non-transitory computer-readable recording medium storing at least one bit stream of visual data, wherein the at least one bit stream of visual data is generated by a method executed by a visual data processing device, wherein the method comprises: determining a residual representation of the visual data based at least on a gain parameter and a first probability distribution parameter of the visual data, wherein the residual representation represents a residual value compared to a second probability distribution representation of the visual data, wherein the gain parameter adjusts a value range of the residual representation; and generating the at least one bit stream based on the residual representation.

[0274] Item 40. A method for storing at least one bitstream of visual data, comprising: determining a residual representation of the visual data based at least on a gain parameter and a first probability distribution parameter of the visual data, the residual representation representing a residual value compared to a second probability distribution representation of the visual data, the gain parameter adjusting a value range of the residual representation; generating the at least one bitstream based on the residual representation; and storing the at least one bitstream in a non-transitory computer-readable recording medium.

[0275] Example Device

[0276] Fig.10 A block diagram of a computing device 1000 in which various embodiments of the present disclosure may be implemented is shown. The computing device 1000 may be implemented as a source device 110 (or data encoder 114 or 200) or a destination device 120 (or data decoder 124 or 300).

[0277] It should be understood that Fig.10 The computing device 1000 shown in FIG. 1 is for illustrative purposes only and is not intended to in any way imply any limitation on the functionality and scope of the embodiments of the present disclosure.

[0278] like Fig.10 As shown, computing device 1000 includes a general computing device 1000. Computing device 1000 may include at least one or more processors or processing units 1010, memory 1020, storage unit 1030, one or more communication units 1040, one or more input devices 1050, and one or more output devices 1060.

[0279] In some embodiments, the computing device 1000 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server provided by a service provider, a large computing device, etc. The user terminal can be, for example, any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet computer, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device, or any combination thereof, and includes accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 1000 can support any type of interface to the user (such as a "wearable" circuit device, etc.).

[0280] The processing unit 1010 may be a physical processor or a virtual processor and may implement various processes based on a program stored in the memory 1020. In a multi-processor system, a plurality of processing units execute computer executable instructions in parallel to improve the parallel processing capability of the computing device 1000. The processing unit 1010 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0281] The computing device 1000 typically includes various computer storage media. Such media can be any media accessible by the computing device 1000, including but not limited to volatile media and non-volatile media, or removable media and non-removable media. The memory 1020 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM) or flash memory) or any combination thereof. The storage unit 1030 can be any removable or non-removable medium, and can include machine-readable media, such as a memory, a flash drive, a disk, or other media that can be used to store information and / or data and can be accessed in the computing device 1000.

[0282] The computing device 1000 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Fig.10 Although not shown in the figure, a magnetic disk drive for reading from and / or writing to a removable nonvolatile magnetic disk and an optical disk drive for reading from and / or writing to a removable nonvolatile optical disk may be provided. In this case, each drive may be connected to the bus (not shown) via one or more data medium interfaces.

[0283] The communication unit 1040 communicates with another computing device via a communication medium. In addition, the functionality of the components in the computing device 1000 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 1000 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general network nodes.

[0284] The input device 1050 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. The output device 1060 may be one or more of various output devices, such as a display, a speaker, a printer, etc. With the help of the communication unit 1040, the computing device 1000 may also communicate with one or more external devices (not shown), such as storage devices and display devices, and the computing device 1000 may also communicate with one or more devices that enable a user to interact with the computing device 1000, or any device that enables the computing device 1000 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.), if necessary. Such communication may be performed via an input / output (I / O) interface (not shown).

[0285] In some embodiments, some or all components of the computing device 1000 may also be arranged in a cloud computing architecture rather than being integrated in a single device. In a cloud computing architecture, components may be provided remotely and work together to implement the functions described in the present disclosure. In some embodiments, cloud computing provides computing, software, data access and storage services, which will not require the end user to know the physical location or configuration of the system or hardware that provides these services. In various embodiments, cloud computing provides services via a wide area network (e.g., the Internet) using a suitable protocol. For example, a cloud computing provider provides applications via a wide area network, which can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding visual data may be stored on a remote server. The computing resources in a cloud computing environment may be merged or distributed at the location of a remote data center. Cloud computing infrastructures may provide services through a shared data center, although they appear as a single access point for users. Therefore, cloud computing architectures may be used to provide components and functions described herein from a service provider at a remote location. Alternatively, they may be provided by a conventional server, or installed directly or otherwise on a client device.

[0286] In an embodiment of the present disclosure, the computing device 1000 may be used to implement visual data encoding / decoding. The memory 1020 may include one or more visual data encoding / decoding modules 1025 having one or more program instructions. These modules can be accessed and executed by the processing unit 1010 to perform the functions of the various embodiments described herein.

[0287] In an example embodiment performing visual data encoding, input device 1050 may receive visual data as input to be encoded 1070. The visual data may be processed by, for example, visual data encoding / decoding module 1025 to generate an encoded bitstream. The encoded bitstream may be provided as output 1080 via output device 1060.

[0288] In an example embodiment where visual data decoding is performed, input device 1050 may receive an encoded bitstream as input 1070. The encoded bitstream may be processed by, for example, visual data codec module 1025 to generate decoded visual data. The decoded visual data may be provided as output 1080 via output device 1060.

[0289] Although the present disclosure has been specifically shown and described with reference to the preferred embodiments of the present disclosure, it will be understood by those skilled in the art that various changes can be made in form and detail without departing from the spirit and scope of the present application as defined by the appended claims. These changes are intended to be encompassed by the scope of the present application. Therefore, the foregoing description of the embodiments of the present application is not intended to be limiting.

Claims

1. A visual data processing method, comprising: determining, for conversion between at least one bitstream of visual data and the visual data, a residual representation of the visual data representing a residual value compared to a second probability distribution representation of the visual data based at least on a gain parameter and a first probability distribution parameter of the visual data, the gain parameter adjusting a value range of the residual representation; as well as The transforming is performed based on the residual representation. 2 . The method according to claim 1 , wherein the at least one bitstream comprises a first bitstream and a second bitstream, the first bitstream comprising arithmetic information of the visual data, and the second bitstream comprising probability distribution information of the visual data.

3. The method of claim 2, wherein the converting comprises decoding the visual data from the first bitstream and the second bitstream, the gain parameters comprising a first gain parameter, Wherein determining the residual representation comprises: determining, based on the second bitstream and a model, the first probability distribution parameters of the visual data; as well as obtaining the residual representation of the visual data based on the first bitstream, the first probability distribution parameter, and the first gain parameter; and The conversion comprises: The visual data is determined based at least on the residual representation of the visual data.

4. The method of claim 3, wherein obtaining the residual representation of the visual data comprises: determining a first intermediate representation of the residual representation based on the first bitstream, the first probability distribution parameters, and a first entropy model; as well as The residual representation is determined by multiplying the first gain parameter by the first intermediate representation.

5. The method of claim 4, wherein the first intermediate representation comprises a plurality of components associated with a plurality of channels of the first entropy model, and the first gain parameter comprises a plurality of gain components, the number of the plurality of gain components being the same as the number of the plurality of channels.

6. The method according to any one of claims 3 to 5, wherein determining the visual data comprises: determining, based on a prediction module and the first sample of the visual data, the second probability distribution representation of the visual data; determining a second sample of the visual data based on the residual representation and the second probability distribution representation of the visual data; and The visual data is reconstructed based on the first sample, the second sample and a synthesis model.

7. The method of claim 6, wherein determining the second probability distribution representation comprises: Determining context information of the visual data based on the first sample and a context module; determining super information of the visual data based at least on the second bitstream and the second entropy model; as well as Based on the context information, the super information and the prediction module, the second probability distribution representation is determined. The method of claim 7 , wherein the super information is further determined based on a super codec.

9. The method according to any one of claims 3 to 8, wherein determining the first probability distribution parameter of the visual data comprises: determining a variance value of the visual data based on the second bitstream and the model; as well as The first probability distribution parameter is determined based on the variance value and a first predefined value, the first predefined value being added to or multiplied by the variance value.

10. The method according to any one of claims 3 to 9, wherein obtaining the residual representation comprises: determining a second intermediate representation of the visual data based on the first bitstream, the first probability distribution parameter, and the first gain parameter; as well as The residual representation is determined based on the second intermediate representation and a second predefined value, which is added to or multiplied by the second intermediate representation.

11. The method of any one of claims 3-10, wherein the model comprises a superscale codec, the superscale codec being not autoregressive.

12. The method of any one of claims 3-11, wherein the residual representation of the visual data comprises a luma residual representation and a chroma residual representation.

13. The method of claim 12, wherein the visual data comprises a luma component and a chroma component, and wherein determining the visual data based at least on the residual representation of the visual data comprises: determining the luma component based at least on the luma residual representation; as well as The chroma components are determined based at least on the chroma residual representation.

14. The method of claim 13, wherein determining the chrominance component comprises: The chroma components are determined based on the chroma residual representation and the luma component.

15. The method of claim 14, wherein a neural network based post-processing module is applied to the luma component, and the chroma components are determined based on the post-processed luma component.

16. The method of claim 15, wherein the neural network based post-processing module comprises an Inter-Channel Correlation Information (ICCI) codec, and the chrominance components are resampled by the ICCI codec.

17. The method according to any one of claims 12-16, wherein the luma residual representation is determined based on a first luma probability distribution parameter and the first gain parameter. The method of claim 17 , wherein the first brightness probability distribution parameter is determined based on a first superscale codec.

19. The method of any one of claims 13-18, wherein the luma component is determined based on the luma residual representation, a first context model, and a first prediction module.

20. The method of claim 19, wherein the luma component is further based on a first supercodec.

21. The method according to any one of claims 12-20, wherein the chroma residual representation is determined based on a first chroma probability distribution parameter and the first gain parameter.

22. The method of claim 21, wherein the first chroma probability distribution parameters are determined based on a second superscale codec.

23. The method of any one of claims 13-22, wherein the chroma component is determined based on the chroma residual representation, a second context model, and a second prediction module.

24. The method of claim 23, wherein the chrominance component is also based on a second supercodec.

25. The method according to any one of claims 13 to 24, further comprising: applying an adaptive color transform to the luma component and the chroma components; as well as The visual data is determined based on the application.

26. The method according to any one of claims 13 to 25, further comprising: At least one of the luma component and the chroma component is updated based on a third predefined value, the third predefined value being added to or multiplied by the at least one of the luma component or the chroma component.

27. The method of claim 2, wherein the converting comprises encoding the visual data into the first bitstream and the second bitstream, the gain parameter comprises a second gain parameter that is the inverse of the first gain parameter, the first gain parameter being associated with a decoding conversion, and The conversion comprises: The first bitstream and the second bitstream are determined based on at least the second gain parameter and the residual representation.

28. The method of any one of claims 1-26, wherein the residual representation comprises a quantized residual representation.

29. A method according to any one of claims 1-28, wherein the gain parameters include multiple gain vectors, a gain vector in the multiple gain vectors has multiple gain components, the visual data includes multiple components associated with multiple channels, and the number of the multiple gain components is the same as the number of the multiple channels.

30. The method of claim 29, wherein a third gain vector is determined based on a convolution of a first gain vector and a second gain vector of the plurality of gain vectors.

31. The method according to any one of claims 1-30, wherein first information on applying the method is included in the bitstream.

32. The method according to any one of claims 1 to 30, further comprising: First information on applying the method is determined based on the codec information.

33. The method according to claim 32, wherein the codec information comprises at least one of the following: the dimension of the visual data, or The color format of the visual data.

34. The method according to any one of claims 31 to 33, wherein the first information indicates at least one of the following: how to apply the method, or Whether to apply the method.

35. The method of any one of claims 1-34, wherein the converting comprises encoding the visual data into the at least one bitstream.

36. The method of any one of claims 1-34, wherein the converting comprises decoding the visual data from the at least one bitstream.

37. A visual data processing device comprising a processor and a non-volatile memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1-36.

38. A non-transitory computer-readable storage medium storing instructions, the instructions causing a processor to execute the method according to any one of claims 1-36.

39. A non-transitory computer-readable recording medium storing at least one bit stream of visual data, wherein the at least one bit stream of visual data is generated by a method performed by a visual data processing device, wherein the method comprises: determining a residual representation of the visual data based at least on a gain parameter and a first probability distribution parameter of the visual data, the residual representation representing a residual value compared to a second probability distribution representation of the visual data, the gain parameter adjusting a value range of the residual representation; as well as The at least one bitstream is generated based on the residual representation.

40. A method for storing at least one bitstream of visual data, comprising: determining a residual representation of the visual data based at least on a gain parameter and a first probability distribution parameter of the visual data, the residual representation representing a residual value compared to a second probability distribution representation of the visual data, the gain parameter adjusting a value range of the residual representation; generating the at least one bitstream based on the residual representation; as well as The at least one bitstream is stored in a non-transitory computer-readable recording medium.

Images