Video image decoding method and device

Abstract

The present application provides a method and apparatus for decoding video images. It relates to the field of video or image compression technology based on artificial intelligence (AI), and more specifically to the field of video compression technology based on neural networks. The method comprises: obtaining a plurality of coefficients according to a compressed code stream of data to be decoded, the plurality of coefficients including a first coefficient; performing probability estimation according to context information of the first coefficient to obtain a first probability distribution; sampling according to the first probability distribution to obtain a first estimated coefficient; and obtaining a reconstructed image according to the first estimated coefficient. Using the solution of the present application to perform multiple decoding on a single compressed code stream can obtain high-quality images with different properties.

Description

Technical Field

[0001] The present application relates to the field of video images, and in particular to a method and device for decoding video images. Background Art

[0002] A digital image is image information recorded as digital signals. A digital image (hereafter referred to as an image) can be viewed as a two-dimensional array with M rows and N columns, consisting of M x N samples. The location of each sample is called a sampling position, and the value of each sample is called a sampling value.

[0003] In applications such as image storage and transmission, image encoding is often required to reduce storage capacity and transmission bandwidth. Image encoding involves two steps: encoding and decoding. A typical encoding process generally includes three steps: transform, quantization, and entropy coding. For an image to be encoded, the first step is to decorrelate the image through transforms to obtain transform coefficients with a more concentrated energy distribution; the second step is to quantize the transform coefficients to obtain quantized coefficients; and the third step is to entropy encode the quantized coefficients to obtain a compressed bitstream. Corresponding to the encoding process, a typical decoding process involves the decoder, after receiving the compressed bitstream, performing three steps: entropy decoding, inverse quantization, and inverse transform to obtain a reconstructed image.

[0004] Since entropy decoding, inverse quantization, and inverse transformation are generally deterministic processes in the above decoding process, for a single compressed code stream, only an image of a single nature can usually be decoded after multiple decodings. Summary of the Invention

[0005] This application provides a method and apparatus for decoding video images. By employing this approach, multiple decoding operations on a single compressed bitstream can produce images with different properties. Furthermore, during the decoding process, the probability distribution used for sampling can be adjusted based on user needs, thereby improving the quality of the reconstructed image.

[0006] These and other objects are achieved by the subject matter of the independent claims. Further implementations are apparent from the dependent claims, the detailed description and the accompanying drawings.

[0007] Particular embodiments are outlined in the accompanying independent claims, further embodiments are outlined in the dependent claims.

[0008] According to a first aspect, the present invention relates to a method for decoding a video image. The method is performed by a decoding device. The method comprises: obtaining a plurality of coefficients, including a first coefficient, based on a compressed bitstream of data to be decoded; performing probability estimation based on context information of the first coefficient to obtain a first probability distribution; sampling based on the first probability distribution to obtain first estimated coefficients, and obtaining a reconstructed image based on the first estimated coefficients. For example, the first estimated coefficients may be estimated values ​​of the first coefficients.

[0009] Optionally, the data to be decoded may be an image, an image block, a slice, or any region of an image.

[0010] In one possible design, the multiple coefficients further include a second coefficient, and the method of the present application further includes:

[0011] Based on the context information of the second coefficient and / or the estimated coefficient obtained by sampling, probability estimation is performed to obtain a second probability distribution, wherein the estimated coefficient obtained by sampling includes the first estimated coefficient; sampling is performed according to the second probability distribution to obtain the second estimated coefficient; then obtaining the reconstructed image based on the first estimated coefficient includes: obtaining the reconstructed image based on the first estimated coefficient and the second estimated coefficient.

[0012] It should be noted here that the second estimated coefficients are obtained after the first estimated coefficients.

[0013] During each decoding of the compressed bitstream, a probability estimate is performed on the decoded coefficients. Based on the probability estimate, sampling is performed to obtain the estimated coefficients. The estimated coefficients are then sampled to reconstruct the image. Because the sampling process is random and uncertain, multiple decoding attempts using this method can yield multiple images with different properties, such as the image with the best subjective quality or the image with the best objective quality.

[0014] In one possible design, multiple coefficients are obtained based on a compressed code stream of the data to be decoded, including:

[0015] Performing entropy decoding on the compressed code stream to obtain a plurality of coefficients, where the plurality of coefficients are a plurality of quantized wavelet coefficients; or,

[0016] The compressed code stream is entropy decoded to obtain a plurality of quantized wavelet coefficients; the plurality of quantized wavelet coefficients are inversely quantized to obtain a plurality of coefficients, which are a plurality of reconstructed wavelet coefficients.

[0017] In one possible design, multiple coefficients are obtained based on a compressed code stream of the data to be decoded, including:

[0018] Performing entropy decoding on the compressed code stream to obtain a plurality of coefficients, where the plurality of coefficients are a plurality of quantized offline discrete cosine transform (DCT) coefficients; or,

[0019] The compressed code stream is entropy decoded to obtain a plurality of quantized DCT coefficients; the plurality of quantized DCT coefficients are inversely quantized to obtain a plurality of coefficients, which are a plurality of reconstructed DCT coefficients.

[0020] In one possible design, multiple coefficients are obtained based on a compressed code stream of the data to be decoded, including:

[0021] Entropy decoding is performed on the compressed code stream to obtain a plurality of coefficients, where the plurality of coefficients are a plurality of characteristic coefficients.

[0022] In one possible design, multiple coefficients are obtained based on a compressed code stream of the data to be decoded, including:

[0023] Decoding the compressed code stream to obtain a plurality of coefficients, where the plurality of coefficients are a plurality of initial pixel reconstruction values; or

[0024] The compressed code stream is decoded to obtain a plurality of initial pixel reconstruction values; the plurality of initial pixel reconstruction values ​​are transformed to obtain a plurality of coefficients, where the plurality of coefficients are a plurality of transformed pixel values.

[0025] By obtaining multiple coefficients in the above-mentioned different ways, the decoding method of the present application can be applied to different decoding scenarios, such as wavelet domain, feature domain, DCT domain and pixel domain.

[0026] In one possible design, performing probability estimation based on context information of the first coefficient to obtain a first probability distribution includes:

[0027] Obtaining a probability distribution model of a first coefficient; processing context information of the first coefficient through a first probability estimation network to obtain parameters of the probability distribution model; and obtaining a first probability distribution based on the probability distribution model and the parameters of the probability distribution model;

[0028] or,

[0029] Processing the context information of the first coefficient through a second probability estimation network to obtain a first probability distribution;

[0030] The first probability estimation network and the second probability estimation network are implemented based on neural networks, and the context information of the first coefficient includes part or all of the multiple coefficients and / or part or all of the estimated coefficients that have been sampled.

[0031] Optionally, the above-mentioned probability distribution model can be a Gaussian model, a Laplace model, a mixed Gaussian model or other models; when the above-mentioned probability distribution model is a Gaussian model, the parameters of the probability distribution model include mean and variance; when the above-mentioned probability distribution model is a Laplace model, the parameters of the probability distribution model include position parameters and size parameters.

[0032] Optionally, the above neural network can be a convolutional neural network, a deep neural network, a recurrent neural network or other neural networks.

[0033] Optionally, the structures and parameters of the first probability estimation network and the second probability estimation network are different, or the structures of the first probability estimation network and the second probability estimation network are the same but the parameters are different.

[0034] The first probability distribution can be obtained in the above manner, thereby preparing for subsequent sampling based on the first probability distribution.

[0035] In one possible design, the first probability distribution is a Gaussian distribution, and sampling is performed according to the first probability distribution to obtain the first estimated coefficient, including:

[0036] Obtain a first random number; determine a first reference value based on the first random number, where the first reference value obeys a Gaussian distribution; and determine a first estimation coefficient based on the first reference value and a mean and a variance of a first probability distribution.

[0037] The first random number is a random number uniformly distributed on [0, 1] generated using a linear congruential method.

[0038] Optionally, the first reference value may obey a standard Gaussian distribution, a common Gaussian distribution, an asymmetric Gaussian distribution, a single Gaussian model, a mixed Gaussian model or other Gaussian distributions.

[0039] Because the first random number is random, the first estimated coefficients obtained through sampling are also random, which in turn leads to randomness and uncertainty in the reconstructed image based on the first estimated coefficients. The sampling process is a random and uncertain process; the multiple reconstructed images obtained based on the estimated coefficients obtained through multiple sampling processes have different properties.

[0040] In one possible design, the method of the present application further includes:

[0041] Preprocessing the variance of the first probability distribution to obtain a processed variance;

[0042] Determining a first estimation coefficient according to a first reference value and a mean and a variance of a first probability distribution includes:

[0043] A first estimation coefficient is determined according to the first reference value, the mean of the first probability distribution, and the processed variance.

[0044] In one possible design, the method of the present application further includes: preprocessing the mean of the first probability distribution according to a scaling factor of the first coefficient to obtain a processed mean;

[0045] Determining a first estimation coefficient according to a first reference value and a mean and a variance of a first probability distribution includes:

[0046] A first estimation coefficient is determined according to the first reference value, the variance of the first probability distribution, and the processed mean.

[0047] In one possible design, preprocessing the variance of the first probability distribution to obtain a processed variance includes:

[0048] The variance of the first probability distribution is set to 0 as the processed variance.

[0049] In one possible design, the multiple coefficients are multiple quantized wavelet coefficients, or the multiple coefficients are multiple reconstructed wavelet coefficients, or the multiple coefficients are multiple quantized DCT coefficients, or the multiple coefficients are multiple reconstructed DCT coefficients, or the multiple coefficients are multiple characteristic coefficients, and the variance of the first probability distribution is preprocessed to obtain a processed variance, including:

[0050] The variance of the first probability distribution is preprocessed according to the scaling factor of the first coefficient to obtain a processed variance.

[0051] Similarly, the variance of the second probability distribution can also be preprocessed according to the scaling factor of the second coefficient, where

[0052] The scaling factor of the first coefficient is the same as the scaling factor of the second coefficient; or,

[0053] The scaling factor of the first coefficient is different from the scaling factor of the second coefficient; or

[0054] When the multiple coefficients are multiple quantized wavelet coefficients or multiple reconstructed wavelet coefficients, if the first coefficient and the second coefficient belong to the same subband, the scaling factor of the first coefficient and the scaling factor of the second coefficient are the same; or if the first coefficient and the second coefficient belong to different subbands, the scaling factor of the first coefficient and the scaling factor of the second coefficient are different,

[0055] or,

[0056] When the multiple coefficients are multiple quantized DCT coefficients or multiple reconstructed DCT coefficients, if the first coefficient and the second coefficient belong to the same frequency band, the scaling factor of the first coefficient and the scaling factor of the second coefficient are the same; or if the first coefficient and the second coefficient belong to different frequency bands, the scaling factor of the first coefficient and the scaling factor of the second coefficient are different,

[0057] or,

[0058] When the multiple coefficients are multiple quantization feature coefficients, or the multiple coefficients are multiple reconstruction feature coefficients, if the first coefficient and the second coefficient belong to the same channel, the scaling factor of the first coefficient and the scaling factor of the second coefficient are the same; or if the first coefficient and the second coefficient belong to different channels, the scaling factor of the first coefficient and the scaling factor of the second coefficient are different.

[0059] Among them, if

[0060] In one possible design, the multiple coefficients are multiple initial pixel reconstruction values, or the multiple pixels are multiple transformed pixel values, and the variance of the first probability distribution is preprocessed to obtain a processed variance, including:

[0061] Preprocessing the variance of the first probability distribution according to a scaling factor of the first coefficient to obtain a processed variance,

[0062] The scaling factor of the first coefficient is the same as the scaling factor of the second coefficient, or the scaling factor of the first coefficient is different from the scaling factor of the second coefficient.

[0063] By preprocessing the first probability distribution, reconstructed images with different properties can be obtained according to user needs, thereby improving the quality of the reconstructed images. For example, by setting the variance of the first probability distribution to 0 as the processed variance, a reconstructed image with optimal signal quality (optimal objective quality) can be obtained, that is, increasing the peak signal-to-noise ratio (PSNR) of the image or reducing the mean-square error (MSE); by setting the scaling factors of multiple coefficients to be the same, an image with optimal subjective quality can be obtained, that is, reducing the PSNR of the image or increasing the MSE of the image; by setting the scaling factors of coefficients belonging to the same part of the image to be the same and setting the scaling factors of coefficients belonging to different parts to be different, an image with properties between the optimal subjective quality and the optimal objective quality can be obtained.

[0064] In one possible design, the multiple coefficients are multiple quantized wavelet coefficients, or the multiple coefficients are multiple reconstructed wavelet coefficients, and obtaining a reconstructed image according to the first estimated coefficients and the second estimated coefficients includes:

[0065] Perform inverse wavelet transform on the first estimation coefficient and the second estimation coefficient to obtain a reconstructed image.

[0066] In one possible design, obtaining a reconstructed image according to the first estimated coefficient and the second estimated coefficient includes:

[0067] When the multiple coefficients are multiple quantized DCT coefficients, performing inverse quantization and inverse DCT on the first estimated coefficients and the second estimated coefficients to obtain a reconstructed image, or,

[0068] When the multiple coefficients are multiple reconstructed DCT coefficients, inverse DCT is performed on the first estimated coefficients and the second estimated coefficients to obtain a reconstructed image.

[0069] In one possible design, the multiple coefficients are multiple transformed pixel values, and obtaining a reconstructed image according to the first estimated coefficients and the second estimated coefficients includes:

[0070] The first estimation coefficient and the second estimation coefficient are inversely transformed to obtain a reconstructed image.

[0071] Since the sampling process is random, the sampling step can be repeated in the present application to obtain multiple reconstructed images. The multiple reconstructed images can be reconstructed images with the best subjective quality or reconstructed images with the best objective quality. The reconstructed image can be used as a reference for intra-frame or inter-frame prediction within the codec loop; it can also be used outside the codec loop to optimize the image quality as a post-processing method. For example: after obtaining multiple reconstructed images through the sampling step and the inverse transformation step, the reconstructed image with the best subjective quality is placed in the image buffer (decoded picture buffer, DPB) or the reference frame set, and used as a reference image for intra-frame or inter-frame prediction within the codec loop; the reconstructed image with the best objective quality is used for post-processing to adjust the subjective quality of the reconstructed image after coding and decoding, so as to improve the quality of the compressed and reconstructed image / video.

[0072] According to a second aspect, the present invention relates to an apparatus for decoding a compressed code stream. The beneficial effects of the apparatus are described in the first aspect and are not further elaborated here. The decoding apparatus has the function of implementing the actions described in the method example of the first aspect. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions described above.

[0073] The method described in the first aspect of the present invention can be performed by the apparatus described in the second aspect of the present invention. Other features and implementations of the method described in the first aspect of the present invention directly depend on the functionality and implementation of the apparatus described in the second aspect of the present invention.

[0074] According to a third aspect, the present invention relates to an apparatus for decoding a video stream, comprising a processor and a memory, wherein the memory stores instructions, and the instructions enable the processor to execute the method according to the first aspect.

[0075] According to a fourth aspect, a computer-readable storage medium is provided, on which instructions are stored, which, when executed, cause one or more processors to encode video data, wherein the instructions cause the one or more processors to perform the method of any possible embodiment of the first aspect.

[0076] According to a fifth aspect, the present invention relates to a computer program product comprising a program code, which, when run, performs the method in any possible embodiment of the first aspect.

[0077] One or more embodiments will be described in detail in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0079] Figure 1 is a block diagram of an example of a video decoding system for implementing an embodiment of the present application;

[0080] Figure 2 is a block diagram of another example of a video decoding system for implementing an embodiment of the present application;

[0081] Figure 3 is a schematic block diagram of a video decoding device for implementing an embodiment of the present application;

[0082] Figure 4 is a schematic block diagram of a video decoding device for implementing an embodiment of the present application;

[0083] Figure 5 A schematic diagram of the structure of a video encoder and decoder provided in an embodiment of the present application;

[0084] Figure 6a This is a schematic diagram of the result after a wavelet transform;

[0085] Figure 6b Schematic diagram of the processing flow of wavelet transform;

[0086] Figure 6c for Figure 6b Schematic diagram of the structure of the deep network used for prediction and update;

[0087] Figure 6d A schematic diagram of the structure of a probability estimation network provided in an embodiment of the present application;

[0088] Figure 6e Schematic diagram of the processing flow of inverse wavelet transform;

[0089] Figure 7 A schematic diagram of a model training provided in an embodiment of the present application;

[0090] Figure 8a A schematic diagram of the structure of another video decoder provided in an embodiment of the present application;

[0091] Figure 8b A schematic diagram of the structure of another video decoder provided in an embodiment of the present application;

[0092] Figure 9A schematic diagram of the structure of another video decoder provided in an embodiment of the present application;

[0093] Figure 10a A schematic diagram of the structure of another video decoder provided in an embodiment of the present application;

[0094] Figure 10b A schematic diagram of the structure of another video decoder provided in an embodiment of the present application;

[0095] Figure 11 A schematic diagram of a decoding process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0096] The embodiments of the present application provide an AI-based video image compression technology, in particular, a neural network-based video compression technology, and specifically provide a decoding method based on probability distribution and sampling to improve the traditional hybrid video encoding and decoding system.

[0097] Video coding generally refers to the processing of a sequence of images to form a video or video sequence. In the field of video coding, the terms "picture", "frame" or "image" can be used as synonyms. Video coding (or commonly referred to as coding) includes two parts: video encoding and video decoding. Video coding is performed on the source side and generally includes processing (for example, compressing) the original video image to reduce the amount of data required to represent the video image (thereby making it more efficient to store and / or transmit). Video decoding is performed on the destination side and generally includes inverse processing relative to the encoder to reconstruct the video image. The "encoding" of the video image (or commonly referred to as image) involved in the embodiment should be understood as the "encoding" or "decoding" of the video image or video sequence. The encoding part and the decoding part are also collectively referred to as codec (encoding and decoding, CODEC).

[0098] In the case of lossless video coding, the original video image can be reconstructed, that is, the reconstructed video image has the same quality as the original video image (assuming there is no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, further compression is performed through quantization, etc. to reduce the amount of data required to represent the video image, but the decoder side cannot fully reconstruct the video image, that is, the quality of the reconstructed video image is lower or worse than the quality of the original video image.

[0099] Since the embodiments of the present application involve the application of neural networks, in order to facilitate understanding, some nouns or terms used in the embodiments of the present application are explained below, and these nouns or terms are also considered part of the content of the invention.

[0100] (1) Neural Network

[0101] A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0102]

[0103] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal of the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0104] (2) Deep Neural Networks

[0105] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. Based on the location of the different layers, the neural network within a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. Each layer is fully connected, meaning that any neuron in layer i is connected to any neuron in layer i+1.

[0106] Although DNN looks complicated, the work of each layer is actually not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and a() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0107] In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as

[0108] It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0109] (3) Convolutional Neural Networks

[0110] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. It consists of a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each of which can be composed of a rectangular arrangement of neurons. Neurons in the same feature plane share weights, which are referred to as convolution kernels. Shared weights can be understood as extracting image information in a position-independent manner. Convolution kernels can be initialized as matrices of random size, and during CNN training, they can learn to acquire reasonable weights. Furthermore, shared weights have the direct benefit of reducing the number of connections between layers of the CNN, thereby reducing the risk of overfitting.

[0111] (4) Recurrent neural networks (RNNs) are used to process sequence data. In traditional neural network models, the layers are fully connected from the input layer to the hidden layer and then to the output layer, while the nodes within each layer are disconnected. Although this ordinary neural network solves many difficult problems, it is still powerless to solve many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word because the previous and next words in a sentence are not independent. RNN is called a recurrent neural network because the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer disconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN. RNN aims to make machines have the ability to remember like humans. Therefore, the output of RNN needs to rely on the current input information and historical memory information.

[0112] (5) Loss function

[0113] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which is to pre-configure the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss as much as possible.

[0114] (6) Backpropagation algorithm

[0115] Neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during training, reducing the reconstruction error loss of the neural network model. Specifically, the forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial neural network model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0116] In the following embodiment of the decoding system 10, the encoder 20 and the decoder 30 are based on Figures 1 to 3 Provide a description.

[0117] Figure 1 FIG1 is a schematic block diagram of an exemplary decoding system 10, such as a video decoding system 10 (or simply decoding system 10) that can utilize the techniques of the present application. Video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in video decoding system 10 represent devices that can be used to perform various techniques according to various examples described in this application.

[0118] like Figure 1 As shown, the decoding system 10 includes a source device 12 for providing encoded image data 21 such as an encoded image to a destination device 14 for decoding the encoded image data 21.

[0119] The source device 12 includes an encoder 20 , and optionally, may include an image source 16 , a preprocessor (or preprocessing unit) 18 such as an image preprocessor, and a communication interface (or communication unit) 22 .

[0120] The image source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage that stores any of the above images.

[0121] In order to distinguish the processing performed by the pre-processor (or pre-processing unit) 18 , the image (or image data) 17 may also be referred to as a raw image (or raw image data) 17 .

[0122] The preprocessor 18 is configured to receive (raw) image data 17 and preprocess the image data 17 to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color grading, or denoising. It will be appreciated that the preprocessor 18 may be an optional component.

[0123] The video encoder (or encoder) 20 is used to receive the pre-processed image data 19 and provide the encoded image data 21 (hereinafter referred to as Figure 2 etc. for further description).

[0124] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device through the communication channel 13 for storage or direct reconstruction.

[0125] The destination device 14 includes a decoder 30 and, in addition or alternatively, may include a communication interface (or communication unit) 28 , a post-processor (or post-processing unit) 32 , and a display device 34 .

[0126] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is a encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0127] The communication interface 22 and the communication interface 28 can be used to send or receive encoded image data (or encoded data) 21 through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0128] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0129] The communication interface 28 corresponds to the communication interface 22 , and can be used, for example, to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded image data 21 .

[0130] Both the communication interface 22 and the communication interface 28 can be configured as follows Figure 1The unidirectional communication interface or the bidirectional communication interface indicated by the arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0131] The video decoder (or decoder) 30 is used to receive the encoded image data 21 and provide decoded image data (or decoded image data) 31 (hereinafter referred to as Figure 3 etc. for further description).

[0132] The post-processor 32 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data), such as the decoded image, to obtain post-processed image data 33, such as the post-processed image. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color grading, cropping, or resampling, or any other processing for generating the decoded image data 31 for display on a display device 34, etc.

[0133] The display device 34 is configured to receive the post-processed image data 33 and display the image to a user or viewer. The display device 34 may be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen.

[0134] The decoding system 10 further includes a training engine 25. The specific training process implemented by the training engine 25 will be described in detail later and will not be repeated here.

[0135] although Figure 1 The source device 12 and the destination device 14 are shown as independent devices, but the device embodiments may also include both the source device 12 and the destination device 14 or the functions of both the source device 12 and the destination device 14, that is, include both the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions. In these embodiments, the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0136] According to the description, Figure 1It will be apparent to the skilled person that the presence and (exact) division of the different units or functionalities in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application.

[0137] The encoder 20 (eg, video encoder 20) or the decoder 30 (eg, video decoder 30), or both, may be configured to generate a video signal by: Figure 2 The processing circuitry shown may be implemented, for example, by one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video encoding processors, or any combination thereof. The encoder 20 may be implemented by processing circuitry 46 to include reference to Figure 2 The various modules discussed in connection with encoder 20 and / or any other encoder systems or subsystems described herein. Decoder 30 may be implemented by processing circuitry 46 to include reference to Figure 3 The processing circuitry 46 may be used to perform the various operations discussed below. Figure 4 As shown, if part of the technology is implemented in software, the device can store the software instructions in a suitable non-transitory computer-readable storage medium and use one or more processors to execute the instructions in hardware to perform the technology of the present invention. One of the video encoder 20 and the video decoder 30 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as Figure 2 shown.

[0138] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook computer or laptop, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or content distribution server), a broadcast receiving device, a broadcast transmitting device, etc., and may not use or use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.

[0139] In some cases, Figure 1The video decoding system 10 shown is merely exemplary, and the techniques provided herein may be applicable to video encoding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other examples, data is retrieved from a local memory, sent over a network, and so on. A video encoding device may encode data and store the data in a memory, and / or a video decoding device may retrieve data from a memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but only encode data to a memory and / or retrieve and decode data from a memory.

[0140] Figure 2 According to an exemplary embodiment, Figure 2 The video encoder 20 and / or Figure 3 FIG2 is an illustrative diagram of an example of a video decoding system 40 including a video decoder 30. The video decoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video codec implemented by processing circuitry 46), an antenna 42, one or more processors 43, one or more memory stores 44, and / or a display device 45.

[0141] like Figure 2 As shown, imaging device 41, antenna 42, processing circuit 46, video encoder 20, video decoder 30, processor 43, memory storage 44, and / or display device 45 can communicate with each other. In different embodiments, video decoding system 40 can include only video encoder 20 or only video decoder 30.

[0142] In some instances, antenna 42 can be used to transmit or receive an encoded bitstream of video data. Additionally, in some instances, display device 45 can be used to present the video data. Processing circuitry 46 can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. Video decoding system 40 can also include an optional processor 43, which can similarly include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. Furthermore, memory storage 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory). In a non-limiting example, memory storage 44 can be implemented as cache memory. In other instances, processing circuitry 46 can include memory (e.g., cache memory, etc.) for implementing an image buffer, etc.

[0143] In some examples, video encoder 20 implemented by logic circuitry may include an image buffer (e.g., implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video encoder 20 implemented by processing circuitry 46 to implement reference Figure 2 and / or any other encoder system or subsystem described herein. Logic circuits may be used to perform the various operations discussed herein.

[0144] In some examples, video decoder 30 may be implemented in a similar manner by processing circuitry 46 to implement reference Figure 3 The various modules discussed herein may be used with reference to the video decoder 30 and / or any other decoder systems or subsystems described herein. In some examples, the video decoder 30 implemented by logic circuitry may include an image buffer (implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video decoder 30 implemented by processing circuitry 46 to implement the video decoder 30 as described above. Figure 3 and / or the various modules discussed with respect to any other decoder system or subsystem described herein.

[0145] In some examples, antenna 42 may be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream may include data related to the encoded video frames, indicators, index values, mode selection data, etc., as discussed herein, such as data related to encoding partitions (e.g., transform coefficients or quantized transform coefficients, optional indicators (as discussed), and / or data defining encoding partitions). Video decoding system 40 may also include video decoder 30 coupled to antenna 42 and configured to decode the encoded bitstream. Display device 45 is configured to present the video frames.

[0146] It should be understood that for the examples described herein with reference to video encoder 20, video decoder 30 can be configured to perform the reverse process. With respect to signaling syntax elements, video decoder 30 can be configured to receive and parse such syntax elements and decode the associated video data accordingly. In some examples, video encoder 20 can entropy encode the syntax elements into an encoded video bitstream. In such examples, video decoder 30 can parse such syntax elements and decode the associated video data accordingly.

[0147] For ease of description, embodiments of the present invention are described with reference to the Versatile Video Coding (VVC) reference software or the High-Efficiency Video Coding (HEVC) developed by the Joint Collaboration Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). Those skilled in the art will appreciate that embodiments of the present invention are not limited to HEVC or VVC.

[0148] Figure 3 Schematic diagram of a video decoding device 300 provided in an embodiment of the present invention. The video decoding device 300 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 300 may be a decoder, such as Figure 1 The video decoder 30 in , may also be an encoder, such as Figure 1 The video encoder 20 in.

[0149] The video decoding device 300 includes: an input port 310 (or input port 310) and a receiver unit (Rx) 320 for receiving data; a processor, logic unit, or central processing unit (CPU) 330 for processing data; for example, the processor 330 may be a neural network processor 330; a transmitter unit (Tx) 340 and an output port 350 (or output port 350) for transmitting data; and a memory 360 for storing data. The video decoding device 400 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the input port 310, the receiver unit 320, the transmitter unit 340, and the output port 350 for outputting or receiving optical or electrical signals.

[0150] The processor 330 is implemented in hardware and software. The processor 330 can be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 330 communicates with the input port 310, the receiving unit 320, the transmitting unit 340, the output port 350, and the memory 360. The processor 330 includes a decoding module 370 (e.g., a decoding module 370 based on a neural network NN). The decoding module 370 implements the embodiments disclosed above. For example, the decoding module 370 performs, processes, prepares, or provides various encoding operations. Therefore, the decoding module 370 provides substantial improvements to the functionality of the video decoding device 300 and affects the switching of the video decoding device 300 to different states. Alternatively, the decoding module 370 is implemented by instructions stored in the memory 360 and executed by the processor 330.

[0151] Memory 360 includes one or more disks, tape drives, and solid-state drives and can be used as overflow data storage for storing programs when such programs are selected for execution, and for storing instructions and data read during program execution. Memory 360 can be volatile and / or non-volatile and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0152] Figure 4 A simplified block diagram of an apparatus 400 is provided for an exemplary embodiment. The apparatus 400 may be used as Figure 1 Either or both of the source device 12 and the destination device 14 in .

[0153] Processor 402 in apparatus 400 may be a central processing unit (CPU). Alternatively, processor 402 may be any other type of device or devices, now available or later developed, capable of manipulating or processing information. While the disclosed implementations may be implemented using a single processor, such as processor 402, as shown, using more than one processor provides greater speed and efficiency.

[0154] In one implementation, the memory 404 in the apparatus 400 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 404. The memory 404 may include code and data 406 that are accessed by the processor 402 via a bus 412. The memory 404 may also include an operating system 408 and application programs 410, which include at least one program that allows the processor 402 to perform the methods described herein. For example, the application programs 410 may include applications 1 through N, as well as a video decoding application that performs the methods described herein.

[0155] The apparatus 400 may also include one or more output devices, such as a display 418. In one example, the display 418 may be a touch-sensitive display that combines a display with touch-sensitive elements that can be used to sense touch input. The display 418 may be coupled to the processor 402 via the bus 412.

[0156] Although bus 412 in device 400 is described herein as a single bus, bus 412 may include multiple buses. Furthermore, secondary storage may be directly coupled to other components of device 400 or accessed via a network and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, device 400 may have a variety of configurations.

[0157] Codec and encoding / decoding method of wavelet coefficients

[0158] Figure 5 Schematic block diagram of an example of a video encoder and decoder for implementing the technology of the present application. Figure 5 In the example of , the video encoder 20 includes a wavelet transform unit 202, a quantization unit 204, and an entropy coding unit 206. The video decoder 30 includes an entropy decoding unit 208, a probability estimation unit 212, a sampling unit 214, and an inverse wavelet transform unit 216. Optionally, the video decoder 30 further includes an inverse quantization unit 210. Figure 5 The video codec shown may also be referred to as an end-to-end video codec or a video codec based on an end-to-end video codec.

[0159] Wavelet transform unit 202

[0160] The wavelet transform unit 202 performs N wavelet transforms on the data to be encoded 201 to obtain 3N+1 sub-bands 203 , where each sub-band contains one or more wavelet coefficients.

[0161] Alternatively, the data to be encoded 201 may be an image in YUV444 format, and the three channels are processed separately without utilizing the correlation between the channels. This embodiment is described based on a single-channel signal. It is understood that the solution of this embodiment can be expanded to a method for multi-channel joint processing.

[0162] Performing N wavelet transforms on the coded data 201 can be understood as performing N wavelet transforms on an image block or an image area, which is not limited here. The image area can be an image, a sub-image, a slice, a patch, etc., which is not limited here. Specifically, the image area can be divided using the quadtree-based partitioning method in the existing coding standard, or the image or image area can be divided into image blocks of the same size (for example, divided into image blocks of 8x8 in average). This application takes a wavelet transform as an example, that is, N=1, and will not be repeated later. Performing a wavelet transform on the coded data 201 will yield the following: Figure 6a The four two-dimensional subbands LL1, HL1, LH1, and HH1 are shown, each of which contains one or more wavelet coefficients. LL1 is called an approximation subband and is a low-resolution approximation of the data to be encoded 201; HL1, LH1, and HH1 are called detail subbands and contain high-frequency information of the data to be encoded 201.

[0163] Optionally, the wavelet transform unit 202 may perform wavelet transform on the coded data 201 using traditional wavelet transform, wavelet transform based on deep neural network, or other similar transform methods, which are not specifically limited here.

[0164] For wavelet transform based on deep neural network, we can Figure 6b The flowchart shown performs a wavelet transform. Figure 6b Taking a one-dimensional signal as an example, the process of wavelet transform is described: first, the input signal is sampled and decomposed, generally into odd and even decomposition, to obtain two sampling signals, and then the two sampling signals undergo mutual prediction and update steps to finally obtain two decomposition results, which are called approximate component and detail component respectively. Among them, the prediction and update steps can be repeated multiple times to obtain the final decomposition result, and are not limited to Figure 6b The prediction and update are implemented based on the deep network. Figure 6b a and b in represent scaling parameters, which are used to balance the energy of different components after the prediction and boosting steps.

[0165] To perform a wavelet transform on a two-dimensional image, you need to use Figure 6b The wavelet transform is performed once in one dimension, and then a two-dimensional wavelet transform is obtained. Four sub-bands are obtained. Specifically, after a wavelet transform is performed on an input image with a width of m and a height of n, the following is obtained: Figure 6a Four two-dimensional sub-bands of width m / 2 and height n / 2 are shown.

[0166] for Figure 6a Both "Deep Network-Based Prediction" and "Deep Network-Based Update" can be used Figure 6c The deep network structure shown in the left figure is used for implementation, but the network parameters used in "prediction based on deep network" and "update based on deep network" are different. Figure 6c In the left figure of , “H×W” indicates that the current convolution layer uses a convolution kernel of size H×W, and “ResB” indicates the residual module, such as Figure 6c As shown in the right figure, rule represents the activation function. Of course, "deep network-based prediction" and "deep network-based update" can also be implemented using other neural network structures, and this is not limited here. The network parameters are built into the codec and do not need to be transmitted.

[0167] Quantization unit 204

[0168] The quantization unit 204 quantizes the wavelet coefficients in the sub-band obtained after the wavelet transform to obtain quantized wavelet coefficients 205 .

[0169] Specifically, when quantizing each wavelet coefficient, each subband may be processed according to a preset order 1, and then the wavelet coefficients in the current subband may be quantized according to a preset order 2 to obtain quantized wavelet coefficients, wherein the preset order 1 may be an existing Z-shaped scanning order, for example: LL1→HL1→LH1→HH1. The preset order 2 may be an existing Z-shaped scanning order, a horizontal scanning order, or a vertical scanning order.

[0170] It should be understood that the above-mentioned preset order 1 and preset order 2 are only examples and are not limitations on the application. Of course, other orders are also possible.

[0171] In one example, uniform quantization can be used for quantization. The quantization step size can be optimized during the joint training process, and each joint training model uses a quantization step size. Assume that each wavelet coefficient is denoted as c, and the quantized wavelet coefficient 205 is denoted as The quantization process can be expressed as follows.

[0172]

[0173] Where QP represents the quantization step size, and [·] indicates rounding to the nearest integer.

[0174] Optionally, before quantizing the wavelet coefficients, the wavelet coefficients may be preprocessed to obtain processed wavelet coefficients, which are then quantized. For example, the obtained wavelet coefficients may be subjected to feature extraction using a neural network, and the feature extraction results may be quantized. Processing the wavelet coefficients before quantization allows the decoder to decode a high-quality reconstructed image.

[0175] Entropy coding unit 206

[0176] The entropy coding unit 206 performs entropy coding on the quantized wavelet coefficients 205 to obtain a compressed code stream 217 .

[0177] Specifically, when entropy coding is performed on each quantized wavelet coefficient 205 , each sub-band may be processed according to preset order one, and then entropy coding is performed on the quantized wavelet coefficients 205 in the sub-band according to preset order two to obtain a compressed code stream.

[0178] Performing entropy coding on each quantized wavelet coefficient 205 (referred to as coefficient in the embodiment for ease of description) includes: performing probability estimation on each coefficient to obtain a probability distribution of the coefficient, and then performing entropy coding on the coefficient according to the probability distribution of the coefficient.

[0179] The probability distribution of the coefficients can be determined as follows:

[0180] The probability distribution model of the current coefficient is obtained for modeling, and then the context information of the current coefficient is input into the probability estimation network for processing to obtain the parameters of the above probability distribution model, and the parameters of the probability distribution model are substituted into the probability distribution model to obtain the probability distribution of the current coefficient.

[0181] Optionally, the probability distribution model can be: a Gaussian single model (GSM), an asymmetric Gaussian model, a Gaussian mixture model (GMM), or a Laplace distribution model. The probability estimation network can be implemented based on a deep learning network, such as a recurrent neural network (RNN) and a pixel-by-pixel convolutional neural network (PixelCNN), etc., without limitation herein.

[0182] As an example, when the probability distribution model is a Gaussian model (single Gaussian model, asymmetric Gaussian model, or mixed Gaussian model), the context information of the current coefficient is input into the probability estimation network for processing to obtain the parameters of the Gaussian model, including the mean μ and variance σ; the mean μ and variance σ are input into the probability distribution model used to obtain the probability distribution of the current coefficient.

[0183] As an example, when the probability distribution model is a Laplace distribution model, the context information of the current coefficient is input into the probability estimation network for processing to obtain the parameters of the Laplace distribution model, including the location parameter μ and the scale parameter b; the location parameter μ and the scale parameter b are introduced into the probability distribution model to obtain the probability distribution of the current coefficient.

[0184] As an example, a typical probability estimation network based on PixelCNN is as follows Figure 6d As shown. "H×W" means that the current convolution layer uses a convolution kernel of size H×W, and "ResB" means the residual module (refer to Figure 6c (right figure in the figure), “* / relu” means using the relu activation function after the current layer.

[0185] It should be noted that the context information of the current coefficient includes: the coefficients that have been encoded in the preset area, and the preset area includes the area within the sub-band where the current coefficient is located, or the area outside the sub-band where the current coefficient is located, which is not limited here. Figure 6a For example, when the current coefficient is a coefficient in subband LL1, the coded coefficients in a certain area in subband LL1 can be used as the context information of the current coefficient. When the current coefficient is a coefficient in subband HL1, the coded coefficients in a certain area in subband LL1 or HL1 can be used as the context information of the current coefficient.

[0186] Entropy decoding unit 208

[0187] The entropy decoding unit 208 performs entropy decoding on the compressed code stream 207 to obtain a plurality of quantized wavelet coefficients 209 .

[0188] Corresponding to the entropy encoding process performed by the entropy encoding unit 206, when processing each wavelet coefficient in the compressed code stream 207, each subband can be processed according to the preset order 1, and then the code stream corresponding to the wavelet coefficient in the current subband is entropy decoded according to the preset order 2 to obtain the quantized wavelet coefficient 209. The preset order 1 and the preset order 2 can be the same as those at the encoding end and are not limited here.

[0189] When entropy decoding is performed on the bitstream corresponding to each wavelet coefficient, a probability estimation is first performed on each wavelet coefficient to obtain a probability distribution of the coefficient. Then, based on the probability distribution, entropy decoding is performed on the bitstream corresponding to the wavelet coefficient to obtain the quantized wavelet coefficient 209. The method for performing probability estimation on each coefficient to obtain the probability distribution of the coefficient is the same as that at the encoding end and will not be repeated here.

[0190] Dequantization unit 210

[0191] The inverse quantization unit 210 inverse quantizes the plurality of quantized wavelet coefficients 209 to obtain a plurality of reconstructed wavelet coefficients 211 .

[0192] Specifically, when dequantizing each quantized wavelet coefficient 209, each subband may be processed according to preset order 1, and then the quantized wavelet coefficients 209 in the current subband may be dequantized according to preset order 2 to obtain reconstructed wavelet coefficients 211. Specifically, the quantized wavelet coefficients 209 are multiplied by the corresponding quantization step size to obtain the reconstructed wavelet coefficients 211. The quantization step size may be a QP. Preset order 1 and preset order 2 may be the same as those at the encoding end and are not limited here.

[0193] It should be noted that, for the video decoder 30, the inverse quantization unit 210 is optional. Figure 5 Indicated by dotted line.

[0194] Probability estimation unit 212

[0195] The input data to the probability estimation unit 212 may be a plurality of quantized wavelet coefficients or a plurality of reconstructed wavelet coefficients. For ease of description, the data input to the probability estimation unit 212 is referred to as a plurality of coefficients. The function of the probability distribution unit 212 is described using the first coefficient and the second coefficient among the plurality of coefficients as an example.

[0196] The probability estimation unit 212 performs probability estimation based on the context information of the first coefficient to obtain a first probability distribution 213; and performs probability estimation based on the context information of the second coefficient and / or the estimated coefficient obtained by sampling to obtain a second probability distribution 213, wherein the estimated coefficient obtained by sampling includes the first estimated coefficient, and the first estimated coefficient is obtained before the second estimated coefficient.

[0197] Specifically, the probability estimation unit 212 performs probability estimation based on the context information of the first coefficient to obtain a first probability distribution 213, including:

[0198] Obtaining a probability distribution model of a first coefficient; processing context information of the first coefficient through a first probability estimation network to obtain parameters of the probability distribution model; and obtaining a first probability distribution based on the probability distribution model and the parameters of the probability distribution model;

[0199] or,

[0200] Processing the context information of the first coefficient through a second probability estimation network to obtain a first probability distribution;

[0201] The first probability estimation network and the second probability estimation network are implemented based on neural networks, and the context information of the first coefficient includes part or all of the multiple coefficients and / or part or all of the estimated coefficients that have been sampled.

[0202] Further, when the data input into the probability estimation unit 212 is a quantized wavelet coefficient, the context information of the first coefficient may include the quantized wavelet coefficient in the first area and the estimated coefficient in the second area, wherein the first area is any area within the subband where the first coefficient in the quantized wavelet coefficient map is located, and the second area is any area within the subband where the first coefficient in the estimated coefficient map is located; when the data input into the probability estimation unit 212 is a reconstructed wavelet coefficient, the context information of the first coefficient may include the reconstructed wavelet coefficient in the first area and the estimated coefficient in the second area, wherein the first area is any area within the subband where the first coefficient in the reconstructed wavelet coefficient map is located, and the second area is any area within the subband where the first coefficient in the estimated coefficient map is located.

[0203] It should be understood that the quantized wavelet coefficient map is an image composed of the multiple quantized wavelet coefficients, the reconstructed wavelet coefficient map is an image composed of the multiple reconstructed wavelet coefficients, and the estimated coefficient map is an image composed of multiple sampled estimated coefficients.

[0204] For the second area, Figure 6a For illustration, when the first coefficient is within subband LL1, the second region may be any region within subband LL1; when the first coefficient is within HL1, the second region may be any region within subband LL1 or HL1.

[0205] For the second coefficient, the second probability distribution may be determined in the above manner, or in the following manner:

[0206] The probability estimation unit 212 performs probability estimation based on the context information of the second coefficient and / or the sampled estimated coefficient to obtain a second probability distribution 213. The sampled estimated coefficient here includes the first estimated coefficient, that is, when performing probability estimation to obtain the second probability distribution, the data input into the third probability estimation network or the fourth probability estimation network below includes the first estimated coefficient.

[0207] Specifically, a probability distribution model of the second coefficient is obtained; context information of the second coefficient and / or the sampled estimated coefficient is processed through a third probability estimation network to obtain parameters of the probability distribution model; and a second probability distribution is obtained based on the probability distribution model and the parameters of the probability distribution model.

[0208] or,

[0209] Processing the context information of the first coefficient through a fourth probability estimation network to obtain a second probability distribution;

[0210] Among them, the third probability estimation network and the fourth probability estimation network are implemented based on neural networks, and the context information of the second coefficient includes part or all of the multiple coefficients and / or part or all of the estimated coefficients that have been sampled.

[0211] It should be noted that the context information of the second coefficient can refer to the relevant description of the context information of the first coefficient, which will not be described again here.

[0212] For the above probability distribution model, please refer to the relevant description in the entropy coding unit 206, which will not be described here. The network structures of the above first probability estimation network, second probability estimation network, third probability estimation network and fourth probability estimation network can be specifically referred to. Figure 6d The network structure shown is not limited here, and other forms of network structures are of course also possible.

[0213] It should be noted that the first probability distribution and the second probability distribution are both output by the probability estimation unit 212 , and are therefore marked with the same identifier 213 .

[0214] Sampling unit 214

[0215] Sampling is performed according to the first probability distribution 213 to obtain the first estimated coefficient 215; sampling is performed according to the second probability distribution 213 to obtain the second estimated coefficient 215. Since the sampling processes are the same for both, the following example uses the first probability distribution 213 as a Gaussian distribution to illustrate how to obtain the first estimated coefficient 215 by sampling according to the first probability distribution 213.

[0216] Specifically, a linear congruential method is used to generate a uniformly distributed random number u on [0,1]; let Then z1 obeys the standard Gaussian distribution. Among them, erf() is the Gaussian error function, which is the cumulative distribution function of the standard normal distribution and is defined as follows:

[0217]

[0218] Let z2 = δ·z1 + μ, then z2 obeys a Gaussian distribution with a mean of μ and a variance of δ, and z2 is the first estimated coefficient 215 mentioned above, where δ and μ are the variance and mean of the first probability distribution 213 mentioned above, respectively.

[0219] Optionally, before sampling, the variance of the first probability distribution 213 is processed. The specific processing process includes: setting the variance of the first probability distribution 213 to 0 as the processed variance; and then sampling according to the processed variance and the mean of the first probability distribution 213 according to the above-mentioned sampling method to obtain the first estimated coefficient 215.

[0220] Optionally, before sampling, the variance of the first probability distribution 213 is processed according to the scaling factor of the first coefficient, and then sampling is performed according to the processed variance and the mean of the first probability distribution 213 in the above sampling manner to obtain the first estimated coefficient 215.

[0221] Optionally, before sampling, the mean of the first probability distribution 213 is processed according to the scaling factor of the first coefficient, and then sampling is performed according to the processed mean and the variance of the first probability distribution 213 in accordance with the above sampling method to obtain the first estimated coefficient 215.

[0222] In one example, when the first probability distribution 213 is a Laplace distribution, sampling is performed according to the first probability distribution 213 to obtain the first estimated coefficient 215, including:

[0223] Generate two uniformly distributed random numbers μ1 and μ2, let z3 = b·log(μ1), z4 = b·log(μ2), and the first estimated coefficient is z5 = z3-z4+μ, where μ and b are the location parameter and scale parameter of the first probability distribution respectively.

[0224] Optionally, before sampling, the scale parameter of the first probability distribution 213 is processed according to the scaling factor of the first coefficient, and then sampling is performed according to the processed scale parameter and the position parameter of the first probability distribution 213 in accordance with the above sampling method to obtain the first estimated coefficient 215.

[0225] Optionally, before sampling, the position parameters of the first probability distribution 213 are processed according to the scaling factor of the first coefficient, and then sampling is performed according to the processed position parameters and the scale parameters of the first probability distribution 213 in accordance with the above sampling method to obtain the first estimated coefficient 215.

[0226] It should be understood that the second estimated coefficient 215 can be obtained according to the second probability distribution 213 in the above manner.

[0227] Optionally, the scaling factor of the first coefficient is the same as the scaling factor of the second coefficient; or, the scaling factor of the first coefficient is different from the scaling factor of the second coefficient; or, if the first coefficient and the second coefficient belong to the same subband, the scaling factor of the first coefficient is the same as the scaling factor of the second coefficient; or if the first coefficient and the second coefficient belong to different subbands, the scaling factor of the first coefficient is different from the scaling factor of the second coefficient, that is, the scaling factors of the coefficients belonging to the same subband are the same, and the scaling factors of the coefficients belonging to different subbands are different.

[0228] By preprocessing the first probability distribution, reconstructed images with different properties can be obtained according to user needs. For example, setting the variance of the first probability distribution to 0 as the processed variance can produce a reconstructed image with optimal signal quality (optimal objective quality), that is, increasing the PSNR of the image or reducing the MSE. By setting the scaling factors of multiple coefficients to the same, an image with optimal subjective quality can be obtained, that is, reducing the PSNR of the image or increasing the MSE of the image. By setting the scaling factors of coefficients belonging to the same part of the image to the same, and setting the scaling factors of coefficients belonging to different parts to different, an image with properties between the optimal subjective quality and the optimal objective quality can be obtained.

[0229] It should be noted that the first estimation coefficient 215 and the second estimation coefficient 215 are both output by the probability estimation unit 212 , and are therefore marked with the same identifier 215 .

[0230] It should be noted that, if the first coefficient is not the first coefficient among the multiple coefficients, the context information of the first coefficient also includes an estimated coefficient obtained before the first estimated coefficient.

[0231] Inverse wavelet transform unit 216

[0232] The inverse wavelet transform unit 216 performs inverse wavelet transform on multiple estimation coefficients (including the first estimation coefficient and the second estimation coefficient) to obtain a reconstructed image 217.

[0233] Corresponding to the encoding end, the wavelet inverse transform method at the decoding end can use traditional wavelet inverse transform or wavelet inverse transform based on deep network or other similar transform methods, which are not limited here. Optionally, the flowchart of wavelet inverse transform based on deep network is as follows Figure 6e shown. Figure 6e Taking one-dimensional signal as an example, the process of wavelet inverse transform is described: Figure 6bThe forward transformation process shown is the opposite. The approximate component and the detail component are first multiplied by the parameters 1 / a and 1 / b. Then, after mutual updating and prediction steps, two signals are obtained, corresponding to the odd-numbered sampling components and the even-numbered sampling components of the original input signal respectively. Finally, the two signals are combined to obtain the reconstructed signal. For the decomposition results LL1, HL1, LH1, HH1 of the two-dimensional image (refer to Figure 6a ), you need to first use LH1, HH1 in the column direction Figure 6e The process shown in the figure performs a one-dimensional wavelet inverse transform to obtain H; for LL1 and HL1, use Figure 6e The process shown in the figure performs a one-dimensional wavelet inverse transform to obtain L; then L and H are transformed in the row direction using Figure 6e The process shown performs a one-dimensional inverse wavelet transform to obtain a reconstructed image 217 .

[0234] It should be understood that the above-mentioned multiple estimated coefficients constitute a two-dimensional image.

[0235] To obtain an efficient image encoding and decoding model, the encoder 20 and decoder 30 are cascaded and jointly trained. The purpose of the training is to optimize the parameters of the relevant deep network modules used in the encoding and decoding process, including the wavelet forward and inverse transform based on the deep network, the entropy coding based on the deep network, and the probability estimation network based on the deep neural network. Figure 7 A block diagram showing joint training.

[0236] The loss function used is:

[0237]

[0238] Among them, the loss function includes three items: the bit rate given by entropy coding based on the deep network The log-likelihood log q(c) of the wavelet coefficient c obtained by wavelet forward transform on the probability distribution q; the reconstructed sample image obtained by inverse transform using the mean of q The mean square error between the sample image x and the input; λ adjusts the importance between the bit rate and the reconstruction loss. Different λ generates different models for compressing images at different compression ratios.

[0239] Optionally, for the second term of the above loss function, other loss functions may also be used, such as multi-scale structural similarity (MS-SSIM) between the reconstructed sample image and the sample image, deep feature loss, etc.

[0240] It should be pointed out that since a plurality of wavelet coefficients are obtained by performing a wavelet forward transform on the sample image, a plurality of probability distributions are obtained; the means and variances of the plurality of probability distributions are processed separately, such as averaging, to obtain the mean average and the variance average. The probability distribution obtained by the two parameters of the mean average and the variance average is the above-mentioned q(c).

[0241] The above training process is implemented by the training engine 50. The training process includes initialization training and joint training. The initialization training process includes:

[0242] The forward transform and inverse transform of the CDF9 / 7 wavelet are used to replace the wavelet forward transform and inverse transform based on the deep network, and the other things remain unchanged to obtain an initialized codec model; after obtaining the sample image, the training engine 50 trains the initialized codec model based on the sample image until the loss value obtained based on the above loss function converges; at this time, the parameters of the probability estimation network and the parameters of the deep network used for entropy coding are kept unchanged, and the forward transform and inverse transform of the wavelet based on the deep network are used to replace the CDF9 / 7 wavelet forward transform and inverse transform to obtain a joint model; the training engine 50 trains the joint model based on the sample image until the loss value obtained based on the above loss function converges; the training is now completed.

[0243] It should be noted that the specific process of inputting the sample image into the model to obtain the compressed code stream and reconstructing the sample image can refer to the specific process performed by the above-mentioned encoder 20 and decoder 30, and will not be described here.

[0244] It should be noted that the above-mentioned deep networks for wavelet forward transform, wavelet inverse transform, and entropy coding and the probability estimation network for probability estimation are obtained from the third-party device after the third-party device is trained based on the above-mentioned training method.

[0245] Decoding method and decoder based on DCT domain

[0246] Figure 8a Schematic block diagram of an example of a video decoder for implementing the technology of the present application. Figure 8a In the example, the video decoder 30 includes an entropy decoding unit 802, a probability estimation unit 806, a sampling unit 808 and an inverse transform unit 810. Optionally, the video decoder 30 further includes an inverse quantization unit 804. Figure 8a The video decoder shown may also be called an end-to-end video decoder or a video decoder based on an end-to-end video decoder.

[0247] First, let’s introduce how to encode in the DCT domain.

[0248] Obtaining data to be encoded, the data to be encoded including image blocks, specifically comprising: dividing an original image or image region into image blocks of a preset size, wherein the size of the image blocks of the preset size can be 4x4, 8x8, 16x16, 32x32, 64x64, 128x128, and 256x256, etc. As another practicable manner, the original image is divided into one or more image blocks, and the size of the image blocks is not limited. The original image can be divided using a quadtree, binary tree, or ternary tree partitioning method in an existing coding standard (H266, H265, H264, AVS2, or AVS3) to obtain one or more image blocks.

[0249] Perform a DCT on the data to be encoded, generating multiple quantized DCT coefficients. After the DCT, the low-frequency components of the encoded data (i.e., the image block) are concentrated in the upper left corner, while the high-frequency components are distributed in the lower right corner. The coefficient values ​​in the first row and first column represent the direct current (DC) coefficients, i.e., the average value of the image block. The remaining coefficients are alternating current (AC) coefficients. The AC and DC coefficients are quantized to generate the quantized AC and DC coefficients, i.e., the multiple quantized DCT coefficients.

[0250] Entropy coding of multiple quantized DCT coefficients can be performed using one of the following methods, which are not limited here:

[0251] Method 1: Entropy coding of multiple quantized DCT coefficients can be performed using existing methods, such as Huffman coding in JPEG and CABAC coding in HEVC.

[0252] Method 2: First, probability modeling is performed on each quantized DCT coefficient to obtain a probability distribution model. Contextual information of the quantized coefficients is then fed into a probability estimation network to estimate the parameters of the probability distribution model. The parameters of the probability distribution model are substituted into the probability distribution model to obtain the probability distribution of the quantized DCT coefficient. Entropy encoding is then performed on the quantized DCT coefficients according to the probability distribution. The quantized DCT coefficients are entropy encoded according to the above method to obtain a compressed bitstream. The contextual information of the quantized DCT coefficients includes some or all of the encoded quantized DCT coefficients.

[0253] The above probability distribution model may be: a single Gaussian model, an asymmetric Gaussian model, a mixed Gaussian model or a Laplace distribution model, etc., which is not limited here.

[0254] The above probability estimation network can use a deep learning network, such as RNN and PixelCNN, etc., which is not limited here.

[0255] Entropy decoding unit 802

[0256] The entropy decoding unit 802 performs entropy decoding on the compressed code stream to obtain a plurality of quantized DCT coefficients.

[0257] The compressed bitstream includes a plurality of DCT coefficient bitstreams. When entropy decoding is performed on the bitstream corresponding to each DCT coefficient, a probability estimation is first performed on each DCT coefficient to obtain a probability distribution of the coefficient. Then, based on the probability distribution, entropy decoding is performed on the bitstream corresponding to the DCT coefficient to obtain a quantized DCT coefficient 209. The method for performing probability estimation on each DCT coefficient to obtain the probability distribution of the coefficient is the same as that at the encoder end and is not further described here.

[0258] Optionally, the Huffman decoding method in JPEG or the CABAC decoding method in HEVC may be sampled to decode the compressed code stream to obtain a plurality of quantized DCT coefficients.

[0259] Dequantization unit 804

[0260] The inverse quantization unit 804 inverse quantizes the multiple quantized DCT coefficients to obtain multiple reconstructed DCT coefficients.

[0261] Specifically, each quantized DCT coefficient is multiplied by a corresponding quantization step size to obtain a reconstructed wavelet coefficient, where the quantization step size may be QP.

[0262] It should be noted that, for the video decoder 30, the inverse quantization unit 804 is optional. Figure 8a Indicated by dotted line.

[0263] Probability estimation unit 806

[0264] The input data to probability estimation unit 806 can be multiple quantized DCT coefficients or multiple reconstructed DCT coefficients. For ease of description, the data input to probability estimation unit 806 is referred to as multiple coefficients. The function of probability distribution unit 806 is described using the first and second coefficients of the multiple coefficients as an example.

[0265] The probability estimation unit 806 performs probability estimation based on the context information of the first coefficient to obtain a first probability distribution; and performs probability estimation based on the context information of the second coefficient and / or the estimated coefficient obtained by sampling to obtain a second probability distribution, wherein the estimated coefficient obtained by sampling includes the first estimated coefficient, and the first estimated coefficient is obtained before the second estimated coefficient.

[0266] Specifically, the probability estimation unit 806 performs probability estimation based on the context information of the first coefficient to obtain a first probability distribution, including:

[0267] Obtaining a probability distribution model of the first coefficient; processing the context information of the first coefficient through a fifth probability estimation network to obtain parameters of the probability distribution model; and obtaining a first probability distribution based on the probability distribution model and the parameters of the probability distribution model;

[0268] or,

[0269] Processing the context information of the first coefficient through a sixth probability estimation network to obtain a first probability distribution;

[0270] Among them, the fifth probability estimation network and the sixth probability estimation network are implemented based on neural networks, and the context information of the first coefficient includes part or all of the multiple coefficients and / or part or all of the estimated coefficients that have been sampled.

[0271] Further, when the data input into the probability estimation unit 806 is a quantized DCT coefficient, the context information of the first coefficient may include the quantized DCT coefficient in the third area and the estimated coefficient in the fourth area, wherein the third area is any area in the quantized DCT coefficient map; when the data input into the probability estimation unit 806 is a reconstructed DCT coefficient, the context information of the first coefficient may include the reconstructed DCT coefficient in the third area and the estimated coefficient in the fourth area, wherein the third area is any area in the reconstructed DCT coefficient map, and the fourth area is any area in the estimated coefficient map.

[0272] It should be understood that the quantized DCT coefficient map is an image composed of the multiple quantized DCT coefficients, the reconstructed DCT coefficient map is an image composed of the multiple reconstructed wavelet DCTs, and the estimated coefficient map is an image composed of multiple sampled estimated coefficients.

[0273] For the second coefficient, the second probability distribution may be determined in the above manner, or in the following manner:

[0274] The probability estimation unit 806 performs probability estimation based on the context information of the second coefficient and / or the sampled estimated coefficient to obtain a second probability distribution. The sampled estimated coefficient here includes the first estimated coefficient, that is, when performing probability estimation to obtain the second probability distribution, the data input into the seventh probability estimation network or the eighth probability estimation network below includes the first estimated coefficient.

[0275] Specifically, a probability distribution model of the second coefficient is obtained; context information of the second coefficient and / or the sampled estimated coefficient is processed through a seventh probability estimation network to obtain parameters of the probability distribution model; and a second probability distribution is obtained based on the probability distribution model and the parameters of the probability distribution model.

[0276] or,

[0277] Processing the context information of the first coefficient through an eighth probability estimation network to obtain a second probability distribution;

[0278] Among them, the seventh probability estimation network and the eighth probability estimation network are implemented based on neural networks, and the context information of the second coefficient includes part or all of the multiple coefficients and / or part or all of the estimated coefficients that have been sampled.

[0279] It should be noted that the context information of the second coefficient can refer to the relevant description of the context information of the first coefficient, which will not be described again here.

[0280] For the above probability distribution model, please refer to the relevant description in the entropy coding unit 206, which will not be described here. The network structures of the fifth probability estimation network, the sixth probability estimation network, the seventh probability estimation network and the eighth probability estimation network can be specifically referred to. Figure 6d The network structure shown is not limited here, and other forms of network structures are of course also possible.

[0281] Sampling unit 808

[0282] Sampling unit 808 performs sampling based on the first probability distribution to obtain the first estimated coefficient, and performs sampling based on the second probability distribution to obtain the second estimated coefficient. Since the sampling processes for the two are the same, the following example uses the first probability distribution as a Gaussian distribution to illustrate how to obtain the first estimated coefficient by sampling based on the first probability distribution.

[0283] Specifically, a linear congruential method is used to generate a uniformly distributed random number u on [0,1]; let Then z1 obeys the standard Gaussian distribution. Among them, erf() is the Gaussian error function, which is the cumulative distribution function of the standard normal distribution and is defined as follows:

[0284]

[0285] Let z2 = δ·z1 + μ, then z2 obeys a Gaussian distribution with mean μ and variance δ, and z2 is the first estimated coefficient mentioned above, where δ and μ are the variance and mean of the first probability distribution mentioned above, respectively.

[0286] Optionally, before sampling, the variance of the first probability distribution is processed. The specific processing process includes: setting the variance of the first probability distribution to 0 as the processed variance; and then sampling according to the processed variance and the mean of the first probability distribution according to the above sampling method to obtain the first estimated coefficient.

[0287] Optionally, before sampling, the variance of the first probability distribution is processed according to the scaling factor of the first coefficient, and then sampling is performed according to the processed variance and the mean of the first probability distribution in accordance with the above sampling method to obtain the first estimated coefficient.

[0288] Optionally, before sampling, the mean of the first probability distribution is processed according to the scaling factor of the first coefficient, and then sampling is performed according to the processed mean and the variance of the first probability distribution in accordance with the above sampling method to obtain the first estimated coefficient.

[0289] In one example, when the first probability distribution is a Laplace distribution, sampling according to the first probability distribution to obtain the first estimated coefficient includes:

[0290] Generate two uniformly distributed random numbers μ1 and μ2, let z3 = b·log(μ1), z4 = b·log(μ2), and the first estimated coefficient is z5 = z3-z4+μ, where μ and b are the location parameter and scale parameter of the first probability distribution respectively.

[0291] Optionally, before sampling, the scale parameter of the first probability distribution is processed according to the scaling factor of the first coefficient, and then sampling is performed according to the processed scale parameter and the position parameter of the first probability distribution in accordance with the above sampling method to obtain the first estimated coefficient.

[0292] Optionally, before sampling, the position parameters of the first probability distribution are processed according to the scaling factor of the first coefficient, and then sampling is performed according to the processed position parameters and the scale parameters of the first probability distribution in accordance with the above sampling method to obtain the first estimated coefficient.

[0293] It should be understood that the second estimated coefficient can be obtained according to the second probability distribution in the above manner.

[0294] Optionally, the scaling factor of the first coefficient is the same as the scaling factor of the second coefficient; or, the scaling factor of the first coefficient is different from the scaling factor of the second coefficient; or, if the first coefficient and the second coefficient belong to the same frequency band, the scaling factor of the first coefficient is the same as the scaling factor of the second coefficient; or if the first coefficient and the second coefficient belong to different frequency bands, the scaling factor of the first coefficient is different from the scaling factor of the second coefficient, that is, the scaling factors of the coefficients belonging to the same frequency band are the same, and the scaling factors of the coefficients belonging to different frequency bands are different.

[0295] The scaling factor ranges from [0,1].

[0296] It should be pointed out that the frequency band can be understood as a coefficient block (a coefficient block obtained by performing DCT transform on an image block, because DCT transform is in blocks) or as coefficients at the same position in each coefficient block, forming a frequency band.

[0297] By preprocessing the first probability distribution, reconstructed images with different properties can be obtained according to user needs. For example, setting the variance of the first probability distribution to 0 as the processed variance can produce a reconstructed image with optimal signal quality (optimal objective quality), that is, increasing the PSNR of the image or reducing the MSE. By setting the scaling factors of multiple coefficients to the same, an image with optimal subjective quality can be obtained, that is, reducing the PSNR of the image or increasing the MSE of the image. By setting the scaling factors of coefficients belonging to the same part of the image to the same, and setting the scaling factors of coefficients belonging to different parts to different, an image with properties between the optimal subjective quality and the optimal objective quality can be obtained.

[0298] It should be noted that, if the first coefficient is not the first coefficient among the multiple coefficients, the context information of the first coefficient also includes an estimated coefficient obtained before the first estimated coefficient.

[0299] Inverse transform unit 810

[0300] The inverse transform unit 810 performs inverse DCT on a plurality of estimation coefficients (including the first estimation coefficient and the second estimation coefficient) to obtain a reconstructed image.

[0301] Optionally, if the input data of the probability estimation unit 806 is a plurality of quantized DCT coefficients, the plurality of estimated coefficients (including the first coefficient and the second coefficient) are successively subjected to inverse quantization and inverse DCT by the inverse quantization unit 804 and the inverse transform unit 810 to obtain a reconstructed image, such as Figure 8b shown.

[0302] Feature domain-based decoding method and decoder

[0303] Figure 9 Schematic block diagram of an example of a video decoder for implementing the technology of the present application. Figure 9 In the example of , the video decoder 30 includes an entropy decoding unit 902, a probability estimation unit 904, a sampling unit 906, and a reconstruction unit 908; Figure 9 The video decoder shown may also be called an end-to-end video decoder or a video decoder based on an end-to-end video decoder.

[0304] Entropy decoding unit 902

[0305] The entropy decoding unit 902 performs entropy decoding on the compressed code stream to obtain a plurality of reconstructed feature coefficients.

[0306] Specifically, the entropy decoding unit 902 entropy decodes the side information from the compressed code stream. Then based on the side information A probability estimation is performed on each reconstructed feature coefficient to obtain a probability distribution for each reconstructed feature coefficient. The entropy decoding unit 902 entropy decodes multiple reconstructed feature coefficients from the compressed bitstream based on the probability distribution of the reconstructed feature coefficients. The multiple reconstructed feature coefficients can form a reconstructed feature map. The size of the reconstructed feature map can be expressed as CxWxH, where C generally refers to the number of channels, and W and H are the width and height of each channel.

[0307] Among them, side information It is also a kind of feature information, that is, a three-dimensional feature map, which contains fewer feature coefficients than the number of feature elements of the feature map y obtained by extracting features from the encoded data.

[0308] Probability estimation unit 904

[0309] The input data to the probability estimation unit 904 may be multiple quantized feature coefficients or multiple reconstructed feature coefficients. For ease of description, the data input to the probability estimation unit 904 is referred to as multiple coefficients. The function of the probability distribution unit 904 is described using the first and second coefficients of the multiple coefficients as an example.

[0310] The probability estimation unit 904 performs probability estimation based on the context information of the first coefficient to obtain a first probability distribution; and performs probability estimation based on the context information of the second coefficient and / or the estimated coefficient obtained by sampling to obtain a second probability distribution, wherein the estimated coefficient obtained by sampling includes the first estimated coefficient, and the first estimated coefficient is obtained before the second estimated coefficient.

[0311] Specifically, the probability estimation unit 904 performs probability estimation based on the context information of the first coefficient to obtain a first probability distribution, including:

[0312] Obtaining a probability distribution model of the first coefficient; processing the context information of the first coefficient through a ninth probability estimation network to obtain parameters of the probability distribution model; and obtaining a first probability distribution based on the probability distribution model and the parameters of the probability distribution model;

[0313] or,

[0314] Processing the context information of the first coefficient through a tenth probability estimation network to obtain a first probability distribution;

[0315] Among them, the ninth probability estimation network and the tenth probability estimation network are implemented based on neural networks, and the context information of the first coefficient includes part or all of the multiple coefficients and / or part or all of the estimated coefficients that have been sampled.

[0316] Further, when the data input into the probability estimation unit 904 is a quantization feature coefficient, the context information for the first coefficient may include the quantization feature coefficient within the fifth region and the estimation coefficient in the sixth region, wherein the fifth region is any region in the quantization feature coefficient map and the sixth region is any region in the estimation coefficient map.

[0317] It should be understood that the quantized feature coefficient map is an image composed of the plurality of quantized feature coefficients, and the estimated coefficient map is an image composed of a plurality of sampled estimated coefficients.

[0318] For the second coefficient, the second probability distribution may be determined in the above manner, or in the following manner:

[0319] The probability estimation unit 904 performs probability estimation based on the context information of the second coefficient and / or the sampled estimated coefficient to obtain a second probability distribution. The sampled estimated coefficient here includes the first estimated coefficient, that is, when performing probability estimation to obtain the second probability distribution, the data input into the following eleventh probability estimation network or the twelfth probability estimation network includes the first estimated coefficient.

[0320] Specifically, a probability distribution model of the second coefficient is obtained; context information of the second coefficient and / or the sampled estimated coefficient is processed by an eleventh probability estimation network to obtain parameters of the probability distribution model; and a second probability distribution is obtained based on the probability distribution model and the parameters of the probability distribution model.

[0321] or,

[0322] Processing the context information of the first coefficient through a twelfth probability estimation network to obtain a second probability distribution;

[0323] Among them, the eleventh probability estimation network and the twelfth probability estimation network are implemented based on neural networks, and the context information of the second coefficient includes part or all of the multiple coefficients and / or part or all of the estimated coefficients that have been sampled.

[0324] It should be noted that the context information of the second coefficient can refer to the relevant description of the context information of the first coefficient, which will not be described again here.

[0325] For the above-mentioned probability distribution model, please refer to the relevant description in the entropy coding unit 206, which will not be described here. The network structures of the above-mentioned ninth probability estimation network, the tenth probability estimation network, the eleventh probability estimation network and the twelfth probability estimation network can be specifically referred to. Figure 6d The network structure shown is not limited here, and other forms of network structures are of course also possible.

[0326] Sampling unit 906

[0327] Sampling unit 906 performs sampling based on the first probability distribution to obtain the first estimated coefficient, and performs sampling based on the second probability distribution to obtain the second estimated coefficient. Since the sampling processes for the two are the same, the following example uses the first probability distribution as a Gaussian distribution to illustrate how to obtain the first estimated coefficient by sampling based on the first probability distribution.

[0328] Specifically, a linear congruential method is used to generate a uniformly distributed random number u on [0,1]; let Then z1 obeys the standard Gaussian distribution. Among them, erf() is the Gaussian error function, which is the cumulative distribution function of the standard normal distribution and is defined as follows:

[0329]

[0330] Let z2 = δ·z1 + μ, then z2 obeys a Gaussian distribution with mean μ and variance δ, and z2 is the first estimated coefficient mentioned above, where δ and μ are the variance and mean of the first probability distribution mentioned above, respectively.

[0331] Optionally, before sampling, the variance of the first probability distribution is processed. The specific processing process includes: setting the variance of the first probability distribution to 0 as the processed variance; and then sampling according to the processed variance and the mean of the first probability distribution according to the above sampling method to obtain the first estimated coefficient.

[0332] Optionally, before sampling, the variance of the first probability distribution is processed according to the scaling factor of the first coefficient, and then sampling is performed according to the processed variance and the mean of the first probability distribution in accordance with the above sampling method to obtain the first estimated coefficient.

[0333] Optionally, before sampling, the mean of the first probability distribution is processed according to the scaling factor of the first coefficient, and then sampling is performed according to the processed mean and the variance of the first probability distribution in accordance with the above sampling method to obtain the first estimated coefficient.

[0334] In one example, when the first probability distribution is a Laplace distribution, sampling according to the first probability distribution to obtain the first estimated coefficient includes:

[0335] Generate two uniformly distributed random numbers μ1 and μ2, let z3 = b·log(μ1), z4 = b·log(μ2), and the first estimated coefficient is z5 = z3-z4+μ, where μ and b are the location parameter and scale parameter of the first probability distribution respectively.

[0336] Optionally, before sampling, the scale parameter of the first probability distribution is processed according to the scaling factor of the first coefficient, and then sampling is performed according to the processed scale parameter and the position parameter of the first probability distribution in accordance with the above sampling method to obtain the first estimated coefficient.

[0337] Optionally, before sampling, the position parameters of the first probability distribution are processed according to the scaling factor of the first coefficient, and then sampling is performed according to the processed position parameters and the scale parameters of the first probability distribution in accordance with the above sampling method to obtain the first estimated coefficient.

[0338] It should be understood that the second estimated coefficient can be obtained according to the second probability distribution in the above manner.

[0339] Optionally, the scaling factor of the first coefficient is the same as the scaling factor of the second coefficient; or, the scaling factor of the first coefficient is different from the scaling factor of the second coefficient; or, if the first coefficient and the second coefficient belong to the same channel, the scaling factor of the first coefficient is the same as the scaling factor of the second coefficient; or if the first coefficient and the second coefficient belong to different channels, the scaling factor of the first coefficient is different from the scaling factor of the second coefficient, that is, the scaling factors of the coefficients belonging to the same channel are the same, and the scaling factors of the coefficients belonging to different channels are different.

[0340] The scaling factor ranges from [0,1].

[0341] By preprocessing the first probability distribution, reconstructed images with different properties can be obtained according to user needs. For example, setting the variance of the first probability distribution to 0 as the processed variance can produce a reconstructed image with optimal signal quality (optimal objective quality), that is, increasing the PSNR of the image or reducing the MSE. By setting the scaling factors of multiple coefficients to the same, an image with optimal subjective quality can be obtained, that is, reducing the PSNR of the image or increasing the MSE of the image. By setting the scaling factors of coefficients belonging to the same part of the image to the same, and setting the scaling factors of coefficients belonging to different parts to different, an image with properties between the optimal subjective quality and the optimal objective quality can be obtained.

[0342] It should be noted that, if the first coefficient is not the first coefficient among the multiple coefficients, the context information of the first coefficient also includes an estimated coefficient obtained before the first estimated coefficient.

[0343] Using the above method, multiple estimated coefficients can be obtained, which form a reconstructed feature map. This reconstructed feature map can be input into a machine vision task module to perform corresponding machine vision tasks, such as object classification, recognition, and segmentation. It can also be input into the reconstruction unit 908.

[0344] If applied to feature map encoding for multiple machine vision tasks, different sampling methods can be used for multiple different machine tasks to obtain reconstructed feature maps with multiple different properties. Multiple reconstructed feature maps with different properties are input into their respective machine vision task modules to perform corresponding machine tasks.

[0345] Reconstruction Unit 908

[0346] The reconstruction unit 908 processes the reconstructed feature map to obtain a reconstructed image, that is, transforms the reconstructed image from the feature domain to the pixel domain.

[0347] The reconstruction unit 908 can be implemented based on a neural network of any structure, such as a fully connected network, a convolutional neural network, a recurrent neural network, etc. The neural network can adopt a multi-layer deep neural network structure to achieve better estimation effect.

[0348] Pixel-domain based decoding method and decoder

[0349] Figure 10a Schematic block diagram of an example of a video decoder for implementing the technology of the present application. Figure 10a In the example of FIG, the video decoder 30 includes a decoding unit 1002, a probability estimation unit 1004, and a sampling unit 1006; in another example, the video decoder 30 includes a decoding unit 1002, a probability estimation unit 1004, a sampling unit 1006, a transform unit 1008, and an inverse transform unit 1010. Figure 10b As shown; Figure 10a and Figure 10b The video decoder shown may also be called an end-to-end video decoder or a video decoder based on an end-to-end video decoder.

[0350] Decoding unit 1002

[0351] The decoding unit 1002 decodes the compressed code stream, such as JPEG decoding, to obtain an initial reconstructed image, which includes a plurality of initial pixel reconstruction values.

[0352] Transformation unit 1008

[0353] The transformation unit 1008 transforms the initial reconstructed image, that is, transforms a plurality of initial pixel reconstruction values ​​to obtain a plurality of transformed pixel values.

[0354] Optionally, the transformation method adopted by the above-mentioned transformation unit 1008 includes but is not limited to wavelet transformation, DCT or feature extraction.

[0355] Probability estimation unit 1004

[0356] The input data to probability estimation unit 1004 can be multiple initial pixel reconstruction values ​​or multiple transformed pixel values. For ease of description, the data input to probability estimation unit 1004 is referred to as multiple coefficients. The function of probability estimation unit 1006 is described using the first and second coefficients of the multiple coefficients as an example.

[0357] The probability estimation unit 1004 performs probability estimation based on the context information of the first coefficient to obtain a first probability distribution; and performs probability estimation based on the context information of the second coefficient and / or the estimated coefficient obtained by sampling to obtain a second probability distribution, wherein the estimated coefficient obtained by sampling includes the first estimated coefficient, and the first estimated coefficient is obtained before the second estimated coefficient.

[0358] Specifically, the probability estimation unit 1004 performs probability estimation based on the context information of the first coefficient to obtain a first probability distribution, including:

[0359] Obtaining a probability distribution model of the first coefficient; processing the context information of the first coefficient through a thirteenth probability estimation network to obtain parameters of the probability distribution model; and obtaining a first probability distribution based on the probability distribution model and the parameters of the probability distribution model;

[0360] or,

[0361] Processing the context information of the first coefficient through a fourteenth probability estimation network to obtain a first probability distribution;

[0362] Among them, the thirteenth probability estimation network and the fourteenth probability estimation network are implemented based on neural networks, and the context information of the first coefficient includes part or all of the multiple coefficients and / or part or all of the estimated coefficients that have been sampled.

[0363] Further, when the data input into the probability estimation unit 1004 is a plurality of initial pixel reconstruction values, the context information for the first coefficient may include the initial pixel reconstruction values ​​within the seventh region and the estimated coefficients in the eighth region, wherein the seventh region is any region in the initial reconstructed image; when the data input into the probability estimation unit 1004 is a transformed pixel value, the context information for the first coefficient may include the transformed pixel values ​​within the seventh region and the estimated coefficients in the eighth region, wherein the seventh region is any region within the transformed image obtained by transforming the initial reconstructed image, and the eighth region is any region in the estimated coefficient graph.

[0364] It should be understood that the estimated coefficient map is an image composed of multiple sampled estimated coefficients.

[0365] For the second coefficient, the second probability distribution may be determined in the above manner, or in the following manner:

[0366] The probability estimation unit 1004 performs probability estimation based on the context information of the second coefficient and / or the sampled estimated coefficient to obtain a second probability distribution. The sampled estimated coefficient here includes the first estimated coefficient, that is, when performing probability estimation to obtain the second probability distribution, the data input into the following fifteenth probability estimation network or sixteenth probability estimation network includes the first estimated coefficient.

[0367] Specifically, a probability distribution model of the second coefficient is obtained; context information of the second coefficient and / or the sampled estimated coefficient is processed through a fifteenth probability estimation network to obtain parameters of the probability distribution model; and a second probability distribution is obtained based on the probability distribution model and the parameters of the probability distribution model.

[0368] or,

[0369] Processing the context information of the first coefficient through a sixteenth probability estimation network to obtain a second probability distribution;

[0370] Among them, the fifteenth probability estimation network and the sixteenth probability estimation network are implemented based on neural networks, and the context information of the second coefficient includes part or all of the multiple coefficients and / or part or all of the estimated coefficients that have been sampled.

[0371] It should be noted that the context information of the second coefficient can refer to the relevant description of the context information of the first coefficient, which will not be described again here.

[0372] For the above probability distribution model, please refer to the relevant description in the entropy coding unit 206, which will not be described here. The network structures of the above thirteenth probability estimation network, fourteenth probability estimation network, fifteenth probability estimation network and sixteenth probability estimation network can be specifically referred to. Figure 6d The network structure shown is not limited here, and other forms of network structures are of course also possible.

[0373] Sampling unit 1006

[0374] Sampling unit 1006 performs sampling based on the first probability distribution to obtain the first estimated coefficient, and performs sampling based on the second probability distribution to obtain the second estimated coefficient. Since the sampling processes for the two are the same, the following example uses the first probability distribution as a Gaussian distribution to illustrate how to obtain the first estimated coefficient by sampling based on the first probability distribution.

[0375] Specifically, a linear congruential method is used to generate a uniformly distributed random number u on [0,1]; let Then z1 obeys the standard Gaussian distribution. Among them, erf() is the Gaussian error function, which is the cumulative distribution function of the standard normal distribution and is defined as follows:

[0376]

[0377] Let z2 = δ·z1 + μ, then z2 obeys a Gaussian distribution with mean μ and variance δ, and z2 is the first estimated coefficient mentioned above, where δ and μ are the variance and mean of the first probability distribution mentioned above, respectively.

[0378] Optionally, before sampling, the variance of the first probability distribution is processed. The specific processing process includes: setting the variance of the first probability distribution to 0 as the processed variance; and then sampling according to the processed variance and the mean of the first probability distribution according to the above sampling method to obtain the first estimated coefficient.

[0379] Optionally, before sampling, the variance of the first probability distribution is processed according to the scaling factor of the first coefficient, and then sampling is performed according to the processed variance and the mean of the first probability distribution in accordance with the above sampling method to obtain the first estimated coefficient.

[0380] Optionally, before sampling, the mean of the first probability distribution is processed according to the scaling factor of the first coefficient, and then sampling is performed according to the processed mean and the variance of the first probability distribution in accordance with the above sampling method to obtain the first estimated coefficient.

[0381] When the first probability distribution is a Laplace distribution, sampling according to the first probability distribution to obtain a first estimation coefficient includes:

[0382] Generate two uniformly distributed random numbers μ1 and μ2, let z3 = b·log(μ1), z4 = b·log(μ2), and the first estimated coefficient is z5 = z3-z4+μ, where μ and b are the location parameter and scale parameter of the first probability distribution respectively.

[0383] Optionally, before sampling, the scale parameter of the first probability distribution is processed according to the scaling factor of the first coefficient, and then sampling is performed according to the processed scale parameter and the position parameter of the first probability distribution in accordance with the above sampling method to obtain the first estimated coefficient.

[0384] Optionally, before sampling, the position parameters of the first probability distribution are processed according to the scaling factor of the first coefficient, and then sampling is performed according to the processed position parameters and the scale parameters of the first probability distribution in accordance with the above sampling method to obtain the first estimated coefficient.

[0385] It should be understood that the second estimated coefficient can be obtained according to the second probability distribution in the above manner.

[0386] Optionally, the scaling factor of the first coefficient is the same as the scaling factor of the second coefficient; or, the scaling factor of the first coefficient is different from the scaling factor of the second coefficient; or, when the above transformation is to DCT, if the first coefficient and the second coefficient belong to the same frequency band, the scaling factor of the first coefficient is the same as the scaling factor of the second coefficient; or if the first coefficient and the second coefficient belong to different frequency bands, the scaling factor of the first coefficient is different from the scaling factor of the second coefficient, that is, the scaling factors of the coefficients belonging to the same frequency band are the same, and the scaling factors of the coefficients belonging to different frequency bands are different;

[0387] Alternatively, when the transform is a wavelet transform, if the first coefficient and the second coefficient belong to the same subband, the scaling factor of the first coefficient and the scaling factor of the second coefficient are the same; or if the first coefficient and the second coefficient belong to different subbands, the scaling factor of the first coefficient and the scaling factor of the second coefficient are different, that is, the scaling factors of the coefficients belonging to the same subband are the same, and the scaling factors of the coefficients belonging to different subbands are different;

[0388] Alternatively, when the above transformation is feature extraction, if the first coefficient and the second coefficient belong to the same channel, the scaling factor of the first coefficient and the scaling factor of the second coefficient are the same; or if the first coefficient and the second coefficient belong to different channels, the scaling factor of the first coefficient and the scaling factor of the second coefficient are different, that is, the scaling factors of the coefficients belonging to the same channel are the same, and the scaling factors of the coefficients belonging to different channels are different.

[0389] The scaling factor ranges from [0,1].

[0390] By preprocessing the first probability distribution, reconstructed images with different properties can be obtained according to user needs. For example, setting the variance of the first probability distribution to 0 as the processed variance can produce a reconstructed image with optimal signal quality (optimal objective quality), that is, increasing the PSNR of the image or reducing the MSE. By setting the scaling factors of multiple coefficients to the same, an image with optimal subjective quality can be obtained, that is, reducing the PSNR of the image or increasing the MSE of the image. By setting the scaling factors of coefficients belonging to the same part of the image to the same, and setting the scaling factors of coefficients belonging to different parts to different, an image with properties between the optimal subjective quality and the optimal objective quality can be obtained.

[0391] It should be noted that, if the first coefficient is not the first coefficient among the multiple coefficients, the context information of the first coefficient also includes an estimated coefficient obtained before the first estimated coefficient.

[0392] In this manner, multiple estimation coefficients can be obtained. If the input to probability estimation unit 1004 is a plurality of initial pixel reconstruction values, the multiple coefficients are a plurality of reconstructed pixel values, and the multiple reconstructed pixel values ​​constitute a reconstructed image. If the input to probability estimation unit 1004 is a plurality of transformed pixel values, the multiple coefficients are a plurality of transformed pixel reconstruction values, and the plurality of transformed pixel reconstruction values ​​are input to inverse transformation unit 1010.

[0393] Inverse transform unit 1010

[0394] The inverse transformation unit 1010 performs inverse transformation on the multiple transformed pixel reconstruction values ​​to obtain multiple reconstructed pixel values, which constitute a reconstructed image.

[0395] It should be pointed out that, from the above description, it can be seen that the actions performed by the above-mentioned transformation unit 1008, probability estimation unit 1004, sampling unit 1008 and inverse transformation unit 1010 are all based on the decoding results of the decoding unit 1002; therefore, for the present implementation scheme, it can be regarded as being implemented in an ordinary decoder and an auxiliary decoding device, wherein the ordinary decoder implements the functions of the decoding unit 1002, and the auxiliary decoding device implements the functions of the transformation unit 1008, probability estimation unit 1004, sampling unit 1008 and inverse transformation unit 1010.

[0396] As can be seen, the solution of this application performs a probability estimation on the decoded coefficients during each decoding of the compressed bitstream, samples the coefficients based on the probability estimation results, and then samples the estimated coefficients to reconstruct the image. Because the sampling process is random and uncertain, multiple decoding of the same compressed bitstream using this method can produce multiple high-quality images with different properties. For example, the image with the best subjective quality or the image with the best objective quality.

[0397] Figure 11 1 is a flowchart illustrating a decoding method according to an embodiment of the present invention. The process 1100 may be performed by the video decoder 30. The process 1100 is described as a series of steps or operations. It should be understood that the process 1100 may be performed in various orders and / or may occur simultaneously, not limited to the process 1100. Figure 11 The execution order shown.

[0398] like Figure 11 As shown, the encoding method includes:

[0399] S1101. Obtain a plurality of coefficients according to a compressed code stream of data to be decoded, where the plurality of coefficients includes a first coefficient.

[0400] Optionally, the data to be decoded may be an image, an image block, a slice, or any region of an image.

[0401] In one possible design, the multiple coefficients further include a second coefficient, and the method of the present application further includes:

[0402] Based on the context information of the second coefficient and / or the estimated coefficient obtained by sampling, probability estimation is performed to obtain a second probability distribution, wherein the estimated coefficient obtained by sampling includes the first estimated coefficient; sampling is performed according to the second probability distribution to obtain the second estimated coefficient; then obtaining the reconstructed image based on the first estimated coefficient includes: obtaining the reconstructed image based on the first estimated coefficient and the second estimated coefficient.

[0403] It should be noted here that the second estimated coefficients are obtained after the first estimated coefficients.

[0404] In one possible design, multiple coefficients are obtained based on a compressed code stream of the data to be decoded, including:

[0405] Performing entropy decoding on the compressed code stream to obtain a plurality of coefficients, where the plurality of coefficients are a plurality of quantized wavelet coefficients; or,

[0406] The compressed code stream is entropy decoded to obtain a plurality of quantized wavelet coefficients; the plurality of quantized wavelet coefficients are inversely quantized to obtain a plurality of coefficients, which are a plurality of reconstructed wavelet coefficients.

[0407] In one possible design, multiple coefficients are obtained based on a compressed code stream of the data to be decoded, including:

[0408] Performing entropy decoding on the compressed code stream to obtain a plurality of coefficients, where the plurality of coefficients are a plurality of quantized offline discrete cosine transform (DCT) coefficients; or,

[0409] The compressed code stream is entropy decoded to obtain a plurality of quantized DCT coefficients; the plurality of quantized DCT coefficients are inversely quantized to obtain a plurality of coefficients, which are a plurality of reconstructed DCT coefficients.

[0410] In one possible design, multiple coefficients are obtained based on a compressed code stream of the data to be decoded, including:

[0411] Entropy decoding is performed on the compressed code stream to obtain a plurality of coefficients, where the plurality of coefficients are a plurality of characteristic coefficients.

[0412] In one possible design, multiple coefficients are obtained based on a compressed code stream of the data to be decoded, including:

[0413] Decoding the compressed code stream to obtain a plurality of coefficients, where the plurality of coefficients are a plurality of initial pixel reconstruction values; or

[0414] The compressed code stream is decoded to obtain a plurality of initial pixel reconstruction values; the plurality of initial pixel reconstruction values ​​are transformed to obtain a plurality of coefficients, where the plurality of coefficients are a plurality of transformed pixel values.

[0415] By obtaining multiple coefficients in the above-mentioned different ways, the decoding method of the present application can be applied to different decoding scenarios, such as wavelet domain, feature domain, DCT domain and pixel domain.

[0416] S1102: Perform probability estimation based on context information of the first coefficient to obtain a first probability distribution.

[0417] In one possible design, performing probability estimation based on context information of the first coefficient to obtain a first probability distribution includes:

[0418] Obtaining a probability distribution model of a first coefficient; processing context information of the first coefficient through a first probability estimation network to obtain parameters of the probability distribution model; and obtaining a first probability distribution based on the probability distribution model and the parameters of the probability distribution model;

[0419] or,

[0420] Processing the context information of the first coefficient through a second probability estimation network to obtain a first probability distribution;

[0421] The first probability estimation network and the second probability estimation network are implemented based on neural networks, and the context information of the first coefficient includes part or all of the multiple coefficients and / or part or all of the estimated coefficients that have been sampled.

[0422] Optionally, the above-mentioned probability distribution model can be a Gaussian model, a Laplace model, a mixed Gaussian model or other models; when the above-mentioned probability distribution model is a Gaussian model, the parameters of the probability distribution model include mean and variance; when the above-mentioned probability distribution model is a Laplace model, the parameters of the probability distribution model include position parameters and size parameters.

[0423] Optionally, the above neural network can be a convolutional neural network, a deep neural network, a recurrent neural network or other neural networks.

[0424] Optionally, the structures and parameters of the first probability estimation network and the second probability estimation network are different, or the structures of the first probability estimation network and the second probability estimation network are the same but the parameters are different.

[0425] The first probability distribution can be obtained in the above manner, thereby preparing for subsequent sampling based on the first probability distribution.

[0426] S1103. Sampling is performed according to the first probability distribution to obtain a first estimation coefficient.

[0427] In one possible design, the first probability distribution is a Gaussian distribution, and sampling is performed according to the first probability distribution to obtain the first estimated coefficient, including:

[0428] Obtain a first random number; determine a first reference value based on the first random number, where the first reference value obeys a Gaussian distribution; and determine a first estimation coefficient based on the first reference value and a mean and a variance of a first probability distribution.

[0429] The first random number is a random number uniformly distributed on [0, 1] generated using a linear congruential method.

[0430] Optionally, the first reference value may obey a standard Gaussian distribution, a common Gaussian distribution, an asymmetric Gaussian distribution, a single Gaussian model, a mixed Gaussian model or other Gaussian distributions.

[0431] Because the first random number is random, the first estimated coefficients obtained through sampling are also random, which in turn leads to randomness and uncertainty in the reconstructed image based on the first estimated coefficients. The sampling process is a random and uncertain process; the multiple reconstructed images obtained based on the estimated coefficients obtained through multiple sampling processes have different properties.

[0432] In one possible design, the method of the present application further includes:

[0433] Preprocessing the variance of the first probability distribution to obtain a processed variance;

[0434] Determining a first estimation coefficient according to a first reference value and a mean and a variance of a first probability distribution includes:

[0435] A first estimation coefficient is determined according to the first reference value, the mean of the first probability distribution, and the processed variance.

[0436] In one possible design, the method of the present application further includes: preprocessing the mean of the first probability distribution according to a scaling factor of the first coefficient to obtain a processed mean;

[0437] Determining a first estimation coefficient according to a first reference value and a mean and a variance of a first probability distribution includes:

[0438] A first estimation coefficient is determined according to the first reference value, the variance of the first probability distribution, and the processed mean.

[0439] In one possible design, preprocessing the variance of the first probability distribution to obtain a processed variance includes:

[0440] The variance of the first probability distribution is set to 0 as the processed variance.

[0441] In one possible design, the multiple coefficients are multiple quantized wavelet coefficients, or the multiple coefficients are multiple reconstructed wavelet coefficients, or the multiple coefficients are multiple quantized DCT coefficients, or the multiple coefficients are multiple reconstructed DCT coefficients, or the multiple coefficients are multiple characteristic coefficients, and the variance of the first probability distribution is preprocessed to obtain a processed variance, including:

[0442] Preprocessing the variance of the first probability distribution according to a scaling factor of the first coefficient to obtain a processed variance,

[0443] The scaling factor of the first coefficient is the same as the scaling factor of the second coefficient; or,

[0444] The scaling factor of the first coefficient is different from the scaling factor of the second coefficient; or

[0445] When the multiple coefficients are multiple quantized wavelet coefficients or multiple reconstructed wavelet coefficients, if the first coefficient and the second coefficient belong to the same subband, the scaling factor of the first coefficient and the scaling factor of the second coefficient are the same; or if the first coefficient and the second coefficient belong to different subbands, the scaling factor of the first coefficient and the scaling factor of the second coefficient are different,

[0446] or,

[0447] When the multiple coefficients are multiple quantized DCT coefficients or multiple reconstructed DCT coefficients, if the first coefficient and the second coefficient belong to the same frequency band, the scaling factor of the first coefficient and the scaling factor of the second coefficient are the same; or if the first coefficient and the second coefficient belong to different frequency bands, the scaling factor of the first coefficient and the scaling factor of the second coefficient are different,

[0448] or,

[0449] When the multiple coefficients are multiple quantization feature coefficients, or the multiple coefficients are multiple reconstruction feature coefficients, if the first coefficient and the second coefficient belong to the same channel, the scaling factor of the first coefficient and the scaling factor of the second coefficient are the same; or if the first coefficient and the second coefficient belong to different channels, the scaling factor of the first coefficient and the scaling factor of the second coefficient are different.

[0450] In one possible design, the multiple coefficients are multiple initial pixel reconstruction values, or the multiple pixels are multiple transformed pixel values, and the variance of the first probability distribution is preprocessed to obtain a processed variance, including:

[0451] Preprocessing the variance of the first probability distribution according to a scaling factor of the first coefficient to obtain a processed variance,

[0452] The scaling factor of the first coefficient is the same as the scaling factor of the second coefficient, or the scaling factor of the first coefficient is different from the scaling factor of the second coefficient.

[0453] S1104: Obtain a reconstructed image according to the first estimation coefficient.

[0454] In one possible design, the multiple coefficients are multiple quantized wavelet coefficients, or the multiple coefficients are multiple reconstructed wavelet coefficients, and obtaining a reconstructed image according to the first estimated coefficients and the second estimated coefficients includes:

[0455] Perform inverse wavelet transform on the first estimation coefficient and the second estimation coefficient to obtain a reconstructed image.

[0456] In one possible design, obtaining a reconstructed image according to the first estimated coefficient and the second estimated coefficient includes:

[0457] When the multiple coefficients are multiple quantized DCT coefficients, performing inverse quantization and inverse DCT on the first estimated coefficients and the second estimated coefficients to obtain a reconstructed image, or,

[0458] When the multiple coefficients are multiple reconstructed DCT coefficients, inverse DCT is performed on the first estimated coefficients and the second estimated coefficients to obtain a reconstructed image.

[0459] In one possible design, the multiple coefficients are multiple transformed pixel values, and obtaining a reconstructed image according to the first estimated coefficients and the second estimated coefficients includes:

[0460] The first estimation coefficient and the second estimation coefficient are inversely transformed to obtain a reconstructed image.

[0461] As can be seen, the solution of this application performs a probability estimation on the decoded coefficients during each decoding of the compressed bitstream, samples the coefficients based on the probability estimation results, and then samples the estimated coefficients to reconstruct the image. Because the sampling process is random and uncertain, multiple decoding of the same compressed bitstream using this method can produce multiple high-quality images with different properties. For example, the image with the best subjective quality or the image with the best objective quality.

[0462] Those skilled in the art will appreciate that the functions described in conjunction with the various illustrative logic blocks, modules, and algorithm steps disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions described in the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media can include computer-readable storage media, which corresponds to tangible media, such as data storage media, or communication media including any media that facilitates the transfer of computer programs from one place to another (e.g., according to a communication protocol). In this manner, computer-readable media can generally correspond to (1) non-transitory tangible computer-readable storage media, or (2) communication media, such as signals or carrier waves. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in this application. A computer program product can include computer-readable media.

[0463] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are actually directed to non-transitory tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), and Blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0464] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor," as used herein, may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described by the various illustrative logical blocks, modules, and steps described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements.

[0465] The techniques of this application can be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., a chipset). Various components, modules, or units are described herein to emphasize functional aspects of devices for performing the disclosed techniques, but they do not necessarily require implementation by different hardware units. In fact, as described above, the various units may be combined in a codec hardware unit in conjunction with appropriate software and / or firmware, or provided by interoperating hardware units (including one or more processors as described above).

[0466] The above description is merely an exemplary embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for decoding a video image, characterized in that: include: Obtaining a plurality of coefficients according to a compressed code stream of data to be decoded, wherein the plurality of coefficients includes a first coefficient; Performing probability estimation based on context information of the first coefficient to obtain a first probability distribution; Sampling according to the first probability distribution to obtain a first estimated coefficient; Obtaining a reconstructed image according to the first estimation coefficients; The first probability distribution is a Gaussian distribution, and sampling according to the first probability distribution to obtain a first estimation coefficient includes: Get the first random number; determining a first reference value according to the first random number, where the first reference value obeys a Gaussian distribution; The first estimation coefficient is determined according to the first reference value and the mean and variance of the first probability distribution.

2. The method according to claim 1, characterized in that The plurality of coefficients further includes a second coefficient, the method further comprising: Performing probability estimation to obtain a second probability distribution based on context information of the second coefficient and / or the sampled estimated coefficient; wherein the sampled estimated coefficient includes the first estimated coefficient; Sampling according to the second probability distribution to obtain a second estimated coefficient; Then, obtaining the reconstructed image according to the first estimated coefficients includes: The reconstructed image is obtained according to the first estimation coefficient and the second estimation coefficient.

3. The method according to claim 1 or 2, characterized in that The method of obtaining a plurality of coefficients according to the compressed code stream of the data to be decoded includes: Performing entropy decoding on the compressed code stream to obtain the multiple coefficients, where the multiple coefficients are multiple quantized wavelet coefficients; or, The compressed code stream is entropy decoded to obtain a plurality of quantized wavelet coefficients; the plurality of quantized wavelet coefficients are inversely quantized to obtain the plurality of coefficients, which are a plurality of reconstructed wavelet coefficients.

4. The method according to claim 1 or 2, characterized in that The method of obtaining a plurality of coefficients according to the compressed code stream of the data to be decoded includes: Performing entropy decoding on the compressed code stream to obtain the multiple coefficients, where the multiple coefficients are multiple quantized offline cosine transform (DCT) coefficients; or The compressed code stream is entropy decoded to obtain the multiple quantized DCT coefficients; the multiple quantized DCT coefficients are inversely quantized to obtain the multiple coefficients, which are multiple reconstructed DCT coefficients.

5. The method according to claim 1 or 2, characterized in that The method of obtaining a plurality of coefficients according to the compressed code stream of the data to be decoded includes: Entropy decoding is performed on the compressed code stream to obtain the multiple coefficients, where the multiple coefficients are multiple characteristic coefficients.

6. The method according to claim 1 or 2, characterized in that The method of obtaining a plurality of coefficients according to the compressed code stream of the data to be decoded includes: Decoding the compressed code stream to obtain the multiple coefficients, where the multiple coefficients are multiple initial pixel reconstruction values; or The compressed code stream is decoded to obtain a plurality of initial pixel reconstruction values; the plurality of initial pixel reconstruction values ​​are transformed to obtain the plurality of coefficients, where the plurality of coefficients are a plurality of transformed pixel values.

7. The method according to claim 6, characterized in that The performing probability estimation according to the context information of the first coefficient to obtain a first probability distribution includes: Obtaining a probability distribution model of the first coefficient; processing context information of the first coefficient through a first probability estimation network to obtain parameters of the probability distribution model; and obtaining the first probability distribution based on the probability distribution model and the parameters of the probability distribution model; or, Processing the context information of the first coefficient through a second probability estimation network to obtain the first probability distribution; The first probability estimation network and the second probability estimation network are implemented based on neural networks, and the context information of the first coefficient includes part or all of the multiple coefficients and / or part or all of the estimated coefficients that have been sampled.

8. The method according to claim 2, characterized in that The method further comprises: Preprocessing the variance of the first probability distribution to obtain a processed variance; The determining the first estimation coefficient according to the first reference value and the mean and variance of the first probability distribution includes: The first estimation coefficient is determined according to the first reference value, the mean of the first probability distribution, and the processed variance.

9. The method according to claim 8, characterized in that The preprocessing of the variance of the first probability distribution to obtain a processed variance includes: The variance of the first probability distribution is set to 0 as the processed variance.

10. The method according to claim 8, characterized in that When the multiple coefficients are multiple quantized wavelet coefficients, or the multiple coefficients are multiple reconstructed wavelet coefficients, or the multiple coefficients are multiple quantized DCT coefficients, or the multiple coefficients are multiple reconstructed DCT coefficients, or the multiple coefficients are multiple characteristic coefficients, preprocessing the variance of the first probability distribution to obtain a processed variance includes: preprocessing the variance of the first probability distribution according to the scaling factor of the first coefficient to obtain a processed variance, The method further includes: preprocessing the variance of the second probability distribution according to a scaling factor of the second coefficient, wherein the scaling factor of the first coefficient is the same as the scaling factor of the second coefficient; or The scaling factor of the first coefficient and the scaling factor of the second coefficient are different; or When the multiple coefficients are the multiple quantized wavelet coefficients or the multiple reconstructed wavelet coefficients, if the first coefficient and the second coefficient belong to the same subband, the scaling factor of the first coefficient and the scaling factor of the second coefficient are the same; or if the first coefficient and the second coefficient belong to different subbands, the scaling factor of the first coefficient and the scaling factor of the second coefficient are different, or, When the multiple coefficients are the multiple quantized DCT coefficients or the multiple reconstructed DCT coefficients, if the first coefficient and the second coefficient belong to the same frequency band, the scaling factor of the first coefficient and the scaling factor of the second coefficient are the same; or if the first coefficient and the second coefficient belong to different frequency bands, the scaling factor of the first coefficient and the scaling factor of the second coefficient are different, or, When the multiple coefficients are multiple quantization feature coefficients, or the multiple coefficients are multiple reconstruction feature coefficients, if the first coefficient and the second coefficient belong to the same channel, the scaling factor of the first coefficient and the scaling factor of the second coefficient are the same; or if the first coefficient and the second coefficient belong to different channels, the scaling factor of the first coefficient and the scaling factor of the second coefficient are different.

11. The method according to claim 8, wherein when the plurality of coefficients are a plurality of initial pixel reconstruction values ​​or the plurality of coefficients are a plurality of transformed pixel values, the preprocessing of the variance of the first probability distribution to obtain a processed variance comprises: preprocessing the variance of the first probability distribution according to the scaling factor of the first coefficient to obtain a processed variance, The scaling factor of the first coefficient is the same as the scaling factor of the second coefficient, or the scaling factor of the first coefficient is different from the scaling factor of the second coefficient.

12. The method according to claim 11, characterized in that When the multiple coefficients are multiple quantized wavelet coefficients, or the multiple coefficients are multiple reconstructed wavelet coefficients, obtaining the reconstructed image according to the first estimation coefficients and the second estimation coefficients includes: Perform inverse wavelet transform on the first estimation coefficient and the second estimation coefficient to obtain the reconstructed image.

13. The method according to claim 11, characterized in that Obtaining the reconstructed image according to the first estimation coefficient and the second estimation coefficient includes: When the multiple coefficients are multiple quantized DCT coefficients, performing inverse quantization and inverse DCT on the first estimation coefficients and the second estimation coefficients to obtain the reconstructed image, or, When the multiple coefficients are multiple reconstructed DCT coefficients, inverse DCT is performed on the first estimated coefficients and the second estimated coefficients to obtain the reconstructed image.

14. The method according to claim 11, characterized in that When the plurality of coefficients are the plurality of transformed pixel values, obtaining the reconstructed image according to the first estimation coefficient and the second estimation coefficient comprises: Perform inverse transformation on the first estimation coefficient and the second estimation coefficient to obtain the reconstructed image.

15. A decoder, characterized in that: The method comprises a processing circuit for executing the method according to any one of claims 1 to 14.

16. A computer program product, characterized in that The method comprises a program code, which, when executed on a computer or a processor, is used to perform the method according to any one of claims 1 to 14.

17. A decoder, characterized in that: include: one or more processors; A non-transitory computer-readable storage medium is coupled to the processor and stores a program executed by the processor, wherein when the program is executed by the processor, the decoder performs the method according to any one of claims 1 to 14.

18. A non-transitory computer-readable storage medium, characterized in that The invention comprises program codes, which, when executed by a computer device, are used to perform the method according to any one of claims 1 to 14.

Images