Video image encoding and decoding method and related equipment

Abstract

The present application provides a video image encoding and decoding method and related equipment. It relates to the field of video or image compression technology based on artificial intelligence (AI), and specifically to the field of video compression technology based on neural networks. The decoding method includes: performing probability estimation processing on input data through a neural network to obtain the probability distribution of residual values ​​of multiple samples included in the residual of the current image, the input data at least includes the residual of a reference image, and the reference image is a decoded image before decoding the current image; performing arithmetic entropy decoding on the code stream based on the probability distribution of residual values ​​of multiple samples included in the residual of the current image to obtain first entropy decoding data, the first entropy decoding data representing the residual of the current image; based on the residual of the current image, obtaining the reconstructed sample value of the current image. The present application can improve the efficiency and accuracy of encoding and decoding.

Description

Technical Field

[0001] The present application relates to the field of video encoding and decoding, and in particular to a video image encoding and decoding method and related equipment. Background Art

[0002] Video coding (video encoding and decoding) is widely used in digital video applications such as broadcast digital television, video transmission over the Internet and mobile networks, real-time conversation applications such as video chat and video conferencing, DVD and Blu-ray Disc, video content acquisition and editing systems, and security applications in camcorders.

[0003] Even in the case of relatively short films, large amounts of video data are required to describe them, which can create difficulties when the data is sent or otherwise transmitted across networks with limited bandwidth capacity. Therefore, video data is often compressed before being transmitted across modern telecommunications networks. Since memory resources may be limited, the size of the video can also be an issue when storing the video on a storage device. Video compression equipment typically uses software and / or hardware at the source to encode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received by a video decompression device at the destination. With limited network resources and the growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase compression rates with little to no impact on image quality.

[0004] In recent years, the application of deep learning in image and video encoding and decoding has become a growing trend. Deep neural networks offer powerful nonlinear processing capabilities. Currently, the main neural network-based image or residual coding methods generally use neural networks (such as AutoEncoder) to transform the image into a feature domain, then quantize and entropy encode the feature values ​​in the feature domain (for example, using arithmetic entropy coding). The decoder performs entropy decoding and dequantization on the received bitstream to obtain the feature domain values, which are then restored through the decoding network to reconstruct the image. Summary of the Invention

[0005] The embodiments of the present application provide a video image encoding and decoding method and related equipment, which can improve encoding and decoding efficiency and accuracy.

[0006] These and other objects are achieved by the subject matter of the independent claims. Further implementations are apparent from the dependent claims, the detailed description and the accompanying drawings.

[0007] Particular embodiments are outlined in the accompanying independent claims, further embodiments are outlined in the dependent claims.

[0008] Based on the first aspect, the present application relates to a method for decoding a video image. The method is performed by a decoding device and includes:

[0009] A code stream of a current image is received; probability estimation processing is performed on the input data using a trained neural network (also referred to as a probability estimation network) to obtain a probability distribution of residual values ​​of multiple samples included in the residual of the current image, where the input data includes at least a residual of a reference image, where the reference image is a decoded image before decoding the current image; arithmetic entropy decoding is performed on the code stream based on the probability distribution of residual values ​​of multiple samples included in the residual of the current image to obtain first entropy decoded data, where the first entropy decoded data represents the residual of the current image; and reconstructed sample values ​​of the current image are obtained based on the residual of the current image.

[0010] Among them, there is no priority in the execution order of the step "receiving the code stream of the current image" and the step "performing probability estimation processing on the input data through a neural network to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image". The former can be executed first, the latter can be executed first, or they can be executed simultaneously.

[0011] Among them, the reference image includes multiple samples. On the decoding side, the residual of the reference image refers to the multiple differences between the reconstructed values ​​of the multiple samples included in the reference image and the predicted values ​​of the multiple samples; the current image includes multiple samples. The residual of the current image refers to the multiple differences between the reconstructed values ​​of the multiple samples included in the current image and the predicted values ​​of the multiple samples; that is, the residual of the current image and the residual of the reference image mentioned on the decoding side both refer to the reconstructed residual.

[0012] The first entropy decoded data is partial entropy decoded data decoded from the bitstream, and the first entropy decoded data is used to represent the residual of the current image. It should be understood that other entropy decoded data can also be decoded from the middle of the bitstream, and this is not limited.

[0013] When decoding a bitstream, the reference image is decoded first, and the current image is decoded later. For example, during the decoding process of a bitstream, image 1 is decoded first, and image 2 is decoded later. Image 1 is the reference image for image 2.

[0014] Optionally, the reference image and the current image may be consecutive or non-consecutive frames; for example, the reference image may be the n-1th frame and the current image may be the nth frame; or the reference image may be the n-2th frame and the current image may be the nth frame. When the reference image and the current image are non-consecutive frames, the difference between the timestamp of the current image and the timestamp of the reference image is less than a preset threshold, thereby ensuring that the residual of the reference image and the residual of the current image are correlated.

[0015] By using a trained neural network to perform probability estimation processing on the residual values ​​of multiple samples included in the residual of the reference image, a probability distribution of the residual values ​​of the multiple samples included in the residual of the current image is obtained. By utilizing the computing power of the neural network to fully explore the correlation between the residual of the current image and the residual of the reference image, a high-precision probability distribution of the residual values ​​of the multiple samples included in the residual of the current image can be obtained. Furthermore, when the probability distribution of the residual values ​​of the multiple samples included in the residual of the current image is used to perform arithmetic entropy decoding on the code stream, the decoding efficiency and decoding accuracy can be improved.

[0016] In one possible design, the neural network is trained through the following process, and the method of this embodiment further includes:

[0017] The training input data is obtained, where the training input data includes a decoded image before decoding the sample image, and the sample image is a known decoded image. The training input data is subjected to probability estimation processing by a neural network to obtain an estimated probability distribution of residual values ​​of multiple samples included in the residual of the sample image; a loss value is obtained based on the known probability distribution and the estimated probability distribution of the residual values ​​of multiple samples included in the residual of the sample image; when the loss value converges to a first threshold, or the current number of training times of the neural network is greater than or equal to a second threshold, the current neural network is determined to be the neural network corresponding to when the neural network training is completed.

[0018] In one possible design, the input data for training includes:

[0019] The residual of the first reference image includes residual values ​​of a plurality of samples, or;

[0020] residual values ​​of a plurality of samples included in the residual of the first reference image and residual values ​​of a plurality of samples included in the residual of the second reference image, or;

[0021] prediction values ​​of a plurality of samples included in the prediction of the sample image, residual values ​​of a plurality of samples included in the residual of the first reference image, and prediction values ​​of a plurality of samples included in the prediction of the first reference image; or;

[0022] Prediction values ​​of multiple samples included in the prediction of the sample image, residual values ​​of multiple samples included in the residual of the first reference image, prediction values ​​of multiple samples included in the prediction of the first reference image, prediction values ​​of multiple samples included in the prediction of the second reference image, and residual values ​​of multiple samples included in the residual of the second reference image;

[0023] The first reference image and the second reference image are decoded images before the sample image is decoded.

[0024] When training the above-mentioned neural network, data such as the residuals of multiple reference images (for example, the first reference image and the second reference image) are used as training samples, so that the trained neural network can estimate a more accurate probability distribution, which is beneficial to improving decoding efficiency and decoding accuracy when using this probability distribution for decoding.

[0025] In one possible design, the training of the above-mentioned neural network can be performed online or offline. When training is performed online, the decoding device can obtain a high-precision neural network in a timely manner, and then obtain a high-precision probability distribution. When the high-precision probability distribution is used to perform arithmetic entropy decoding on the code stream, the decoding efficiency and decoding accuracy can be improved.

[0026] In one possible design, the above probability distribution represents the probability distribution of multiple difference values ​​between the reconstructed values ​​of multiple samples included in the residual of the current image and the predicted values ​​of the multiple samples; or, represents the probability distribution of the residual values ​​of each sample in the multiple samples of the current image, or represents the probability distribution of the residual values ​​of multiple samples of the current image.

[0027] In one possible design, the first entropy decoded data includes a residual of the current image, or a feature map of the residual of the current image, or a transformed and quantized residual of the current image.

[0028] In one possible design, the first entropy decoded data is a feature map of the residual of the current image, and the decoding method of the present application further includes:

[0029] The feature map of the residual of the current image is passed through the decoding network to obtain the residual of the current image. By transforming the residual of the current image into the feature domain, the feature map of the residual of the current image is first obtained during decoding, and then the feature map of the residual of the current image is passed through the decoding network to obtain the residual of the current image, which is conducive to improving decoding efficiency.

[0030] In one possible design, the first entropy decoded data is a residual of the current image after transformation and quantization, and the decoding method of the present application further includes:

[0031] The residual of the current image after transformation and quantization is dequantized and detransformed to obtain the residual of the current image.

[0032] One possible inverse transform method is the inverse discrete cosine transform (DCT). In addition, the inverse discrete sine transform (DST) or the inverse discrete wavelet transform (DWT) can also be performed.

[0033] In one possible design, the input data further includes super-prior information, which is used to characterize deeper features of the residual of the current image, and can be obtained by extracting features from the feature map of the residual of the current image using a convolutional neural network; the above-mentioned neural network includes a convolutional network, a uniform pooling layer, a first convolutional layer, a splicing layer and a normalized exponential function (Softmax) layer, and the convolutional network includes multiple convolutional layers and an excitation (ReLU) layer, wherein the input data is subjected to probability estimation processing by the neural network to obtain a probability distribution of residual values ​​of multiple samples included in the residual of the current image, including:

[0034] The convolutional network is used to extract the feature information from the residual and super-prior information of the reference image; the feature information extracted by the convolutional network is probabilistically estimated through the uniform pooling layer, the first convolutional layer, the splicing layer and the Softmax layer, and the probability distribution of the residual values ​​of multiple samples included in the residual of the current image is output.

[0035] When estimating the above probability distribution through a neural network, due to the introduction of super-prior information, in order to comprehensively extract features from the super-prior information and other information (such as the residual of the reference image), by introducing the first convolutional layer and the splicing layer, and extracting the feature information of the residual of the reference image and the super-prior information through the convolutional network, and then performing probability estimation on the above-extracted feature information through the uniform pooling layer, the first convolutional layer, the splicing layer and the Softmax layer, a more accurate probability distribution can be obtained. When using this probability distribution to perform arithmetic entropy decoding on the bitstream, it is beneficial to improve decoding efficiency and decoding accuracy.

[0036] In one possible design, the input data also includes a prediction of a reference image and a prediction of a current image; the neural network includes a convolutional network, a uniform pooling layer, and a softmax layer; the convolutional network includes multiple convolutional layers and an excitation ReLU layer, wherein: the neural network performs probability estimation processing on the input data to obtain a probability distribution of residual values ​​of multiple samples of the residual of the current image, including:

[0037] The convolutional network is used to extract the residual of the reference image, the prediction of the reference image, and the feature information in the prediction of the current image; the uniform pooling layer and the Softmax layer are used to perform probability estimation on the feature information extracted by the convolutional network, and the probability distribution of the residual values ​​of multiple samples included in the residual of the current image is output.

[0038] When estimating the above probability distribution through a neural network, the prediction of the reference image and the prediction of the current image are introduced, and the feature information in the residual of the reference image, the prediction of the reference image and the prediction of the current image are extracted through a convolutional network; the probability of the above extracted feature information is estimated through a uniform pooling layer and a softmax layer, and a more accurate probability distribution can be obtained. When using this probability distribution to perform arithmetic entropy decoding on the bitstream, it is beneficial to improve decoding efficiency and decoding accuracy.

[0039] In one possible design, the input data includes residuals of multiple reference images, where the multiple reference images are multiple decoded images before decoding the current image; the neural network includes a convolutional network, a uniform pooling layer, and a softmax layer; the convolutional network includes multiple convolutional layers and ReLU layers, wherein:

[0040] The performing probability estimation processing on the input data by the neural network to obtain the probability distribution of residual values ​​of the plurality of samples included in the residual of the current image includes:

[0041] The feature information of the residuals of multiple reference images is extracted through the convolutional network; the feature information extracted by the convolutional network is probabilistically estimated through the uniform pooling layer and the Softmax layer, and the probability distribution of the residual values ​​of multiple samples included in the residual of the current image is output.

[0042] When estimating the above probability distribution through a neural network, the residuals of multiple reference images are introduced, and the feature information of the residuals of multiple reference images is extracted through a convolutional network; the probability of this feature information is estimated through a uniform pooling layer and a softmax layer, which can obtain a more accurate probability distribution. When using this probability distribution to perform arithmetic entropy decoding on the bitstream, it is beneficial to improve decoding efficiency and decoding accuracy.

[0043] In one possible design, the input data includes residuals of multiple reference images, predictions of multiple reference images, and a prediction of a current image, where the multiple reference images are multiple decoded images before decoding the current image; the neural network includes a convolutional network, a uniform pooling layer, and a Softmax layer; the convolutional network includes multiple convolutional layers and ReLU layers, wherein the neural network performs probability estimation processing on the input data to obtain a probability distribution of residual values ​​of multiple samples included in the residual of the current image, including:

[0044] The convolutional network is used to extract the residuals of multiple reference images, the predictions of multiple reference images, and the predicted feature information of the current image; the uniform pooling layer and the Softmax layer are used to perform probability estimation on the feature information extracted by the convolutional network, and the probability distribution of the residual values ​​of multiple samples included in the residual of the current image is output.

[0045] When estimating the above probability distribution through a neural network, the residuals and predictions of multiple reference images and the prediction of the current image are introduced, and the feature information of the residuals of multiple reference images, the predictions of multiple reference images and the prediction of the current image is extracted through a convolutional network; the probability of this feature information is estimated through a uniform pooling layer and a Softmax layer, and a more accurate probability distribution can be obtained. When using this probability distribution to perform arithmetic entropy decoding on the bitstream, it is beneficial to improve decoding efficiency and decoding accuracy.

[0046] The prediction of the current image is obtained by predicting the current image using reconstructed sample values ​​of a reference image, or by upsampling reconstructed sample values ​​of a low-resolution image of the current image, and the resolution of the sampled image is the same as the resolution of the current image. The low-resolution image of the current image can be obtained by downsampling the current image.

[0047] Similarly, the prediction signal of the reference image is obtained by predicting the reference image using the reconstructed sample values ​​of the image before the reference is decoded; or it is obtained by upsampling the reconstructed sample values ​​of the low-resolution image of the reference image, and the resolution of the image obtained by upsampling the reconstructed image of the low-resolution image of the reference image is the same as the resolution of the reference image. The low-resolution image of the reference image can be obtained by downsampling the reference image.

[0048] Based on the second aspect, the present application relates to a method for encoding a video image. The method is performed by an encoding device, and the method includes:

[0049] Obtain a residual of a current image, where the current image includes multiple samples, and the residual of the current image is multiple differences between original values ​​of the multiple samples and predicted values ​​of the multiple samples; perform probability estimation processing on input data through a trained neural network (also known as a probability estimation network) to obtain a probability distribution of residual values ​​of the multiple samples included in the residual of the current image, where the input data at least includes a residual of a reference image, which is an encoded image before encoding the current image; perform arithmetic entropy coding on first entropy coded data based on the probability distribution of the residual values ​​of the multiple samples included in the residual of the current image to obtain a code stream of the current image, where the first entropy coded data represents the residual of the current image.

[0050] Among them, there is no priority in the execution order of the step "obtaining the residual of the current image" and the step "performing probability estimation processing on the input data through a neural network to obtain the probability distribution of the residual of the current image". The former can be executed first, the latter can be executed first, or they can be executed simultaneously.

[0051] The reference image includes multiple samples. On the encoding side, the residual of the reference image refers to the multiple differences between the reconstructed values ​​of the multiple samples included in the reference image and the predicted values ​​of the multiple samples; that is, the residual of the reference image referred to on the encoding side refers to the reconstructed residual. On the encoding side, the current image includes multiple samples. The residual of the current image refers to the multiple differences between the original values ​​of the multiple samples included in the current image and the predicted values ​​of the multiple samples; that is, the residual of the current image referred to on the encoding side refers to the original residual.

[0052] The prediction of the current image includes prediction values ​​of multiple samples; optionally, the prediction of the current image can be obtained by predicting the current image using reconstructed sample values ​​of a reference image, or by upsampling reconstructed sample values ​​of a low-resolution image of the current image, where the resolution of the sampled image is the same as the resolution of the current image. The low-resolution image of the current image can be obtained by downsampling the current image.

[0053] The reference image is an image that has been encoded before the current image is encoded. This means that the reference image has been encoded before the current image is encoded. For example, when encoding, image 1 is encoded first and then image 2. Image 1 is the reference image for image 2.

[0054] Optionally, the current image and the reference image may be consecutive frames or non-consecutive frames. When the reference image and the current image are non-consecutive frames, the difference between the timestamp of the current image and the timestamp of the reference image is less than a preset threshold, thereby ensuring that the residual of the reference image is correlated with the residual of the current image.

[0055] By performing probability estimation processing on the residual values ​​of multiple samples included in the residual of the reference image through a trained neural network, a probability distribution of the residual values ​​of the multiple samples included in the residual of the current image is obtained. By fully exploiting the computing power of the neural network to fully explore the correlation between the residual of the current image and the residual of the reference image, a high-precision probability distribution of the residual values ​​of the multiple samples included in the residual of the current image can be obtained. Furthermore, when arithmetic entropy coding is performed on the first entropy coded data using the probability distribution of the residual values ​​of the multiple samples included in the residual of the current image, coding efficiency and coding accuracy can be improved.

[0056] In one possible design, the neural network is a trained neural network, and the method of this implementation further includes:

[0057] Acquire training input data, where the training input data includes an encoded image before encoding a sample image, where the sample image is a known encoded image; perform probability estimation processing on the training input data through a neural network to obtain an estimated probability distribution of residual values ​​of multiple samples included in the residual of the sample image; obtain a loss value based on the known probability distribution and the estimated probability distribution of the residual values ​​of multiple samples included in the residual of the sample image; when the loss value converges to a first threshold, or the current number of training times of the neural network is greater than or equal to a second threshold, determine that the current neural network is the neural network corresponding to when the neural network training is completed.

[0058] In one possible design, the input data for training includes:

[0059] The residual of the first reference image includes residual values ​​of a plurality of samples, or;

[0060] residual values ​​of a plurality of samples included in the residual of the first reference image and residual values ​​of a plurality of samples included in the residual of the second reference image, or;

[0061] prediction values ​​of a plurality of samples included in the prediction of the sample image, residual values ​​of a plurality of samples included in the residual of the first reference image, and prediction values ​​of a plurality of samples included in the prediction of the first reference image; or;

[0062] Prediction values ​​of multiple samples included in the prediction of the sample image, residual values ​​of multiple samples included in the residual of the first reference image, prediction values ​​of multiple samples included in the prediction of the first reference image, prediction values ​​of multiple samples included in the prediction of the second reference image, and residual values ​​of multiple samples included in the residual of the second reference image;

[0063] The first reference image and the second reference image are encoded images before the sample image is encoded.

[0064] When training the above-mentioned neural network, data such as the residuals of multiple reference images (a first reference image and a second reference image) are used as training samples, so that the trained neural network can estimate a more accurate probability distribution. When using this probability distribution to perform arithmetic entropy coding on the first entropy coded data, it is beneficial to improve coding efficiency and coding accuracy.

[0065] In one possible design, the training of the above-mentioned neural network can be performed online or offline. When training is performed online, the encoding device can obtain a high-precision neural network in a timely manner, and then obtain a high-precision probability distribution, which is beneficial to improving the encoding efficiency and encoding accuracy when using the probability distribution to perform arithmetic entropy encoding on the first entropy coded data.

[0066] In one possible design, the above probability distribution represents the probability distribution of multiple difference values ​​between the original values ​​of multiple samples included in the residual of the current image and the predicted values ​​of the multiple samples; or, represents the probability distribution of the residual values ​​of each sample point in the multiple samples of the current image; or, represents the probability distribution corresponding to the residual values ​​of multiple samples of the current image.

[0067] In one possible design, the first entropy coded data includes the residual of the current image, or a feature map of the residual of the current image, or the residual of the current image after transformation and quantization.

[0068] In one possible design, the first entropy coded data is a feature map of the residual of the current image. The encoding method of the present application further includes: passing the residual of the current image through a coding network to obtain a feature map of the residual of the current image. By transforming the residual of the current image into a feature domain, encoding efficiency is improved.

[0069] It should be pointed out that the above encoding network can be regarded as a feature extraction network.

[0070] In one possible design, the first entropy coded data is the residual of the current image after transformation and quantization. The encoding method of the present application also includes: transforming and quantizing the residual of the current image to obtain the residual of the current image after transformation and quantization.

[0071] One possible transformation method is DCT, and DST or DWT can also be performed. Quantization can use methods such as uniform quantization or non-uniform quantization.

[0072] In one possible design, in order to improve the coding accuracy, the input data further includes super-prior information, which is used to characterize the deeper features of the current image. The feature map of the residual of the current image can be extracted using a convolutional neural network. The above-mentioned neural network includes a convolutional network, a uniform pooling layer, a first convolutional layer, a splicing layer and a normalized exponential function (Softmax) layer. The convolutional network includes multiple convolutional layers and an excitation (ReLU) layer. The input data is subjected to probability estimation processing by the neural network to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image, including:

[0073] The convolutional network is used to extract the feature information from the residual and super-prior information of the reference image; the feature information extracted by the convolutional network is probabilistically estimated through the uniform pooling layer, the first convolutional layer, the splicing layer and the Softmax layer, and the probability distribution of the residual values ​​of multiple samples included in the residual of the current image is output.

[0074] When estimating the above probability distribution through a neural network, due to the introduction of super-prior information, in order to comprehensively extract features from the super-prior information and other information (such as the residual of the reference image), a first convolutional layer and a splicing layer are introduced. Then, the feature information of the residual of the reference image and the super-prior information is extracted through the convolutional network. The probability of the extracted feature information is estimated through the uniform pooling layer, the first convolutional layer, the splicing layer and the Softmax layer. A more accurate probability distribution can be obtained. When the first entropy coded data is arithmetic entropy coded using this probability distribution, it is beneficial to improve coding efficiency and coding accuracy.

[0075] In one possible design, the input data also includes a prediction of a reference image and a prediction of a current image; the neural network includes a convolutional network, a uniform pooling layer, and a softmax layer; the convolutional network includes multiple convolutional layers and an excitation ReLU layer, wherein: the neural network performs probability estimation processing on the input data to obtain a probability distribution of residual values ​​of multiple samples of the residual of the current image, including:

[0076] The convolutional network is used to extract the residual of the reference image, the prediction of the reference image, and the feature information in the prediction of the current image; the uniform pooling layer and the Softmax layer are used to perform probability estimation on the feature information extracted by the convolutional network, and the probability distribution of the residual values ​​of multiple samples included in the residual of the current image is output.

[0077] When estimating the above probability distribution through a neural network, the prediction of the reference image and the prediction of the current image are introduced, and the feature information in the residual of the reference image, the prediction of the reference image and the prediction of the current image is extracted through a convolutional network; the above extracted feature information is probability estimated through a uniform pooling layer and a Softmax layer, and a more accurate probability distribution can be obtained. When using this probability distribution to perform arithmetic entropy coding on the first entropy coded data, it is beneficial to improve coding efficiency and coding accuracy.

[0078] In one possible design, input data includes residuals of multiple reference images, where the multiple reference images are multiple encoded images before encoding the current image; the neural network includes a convolutional network, a uniform pooling layer, and a softmax layer; the convolutional network includes multiple convolutional layers and ReLU layers, wherein:

[0079] The performing probability estimation processing on the input data by the neural network to obtain the probability distribution of residual values ​​of the plurality of samples included in the residual of the current image includes:

[0080] When estimating the above-mentioned probability distribution through a neural network, the residuals of multiple reference images are introduced, and the feature information of the residuals of the multiple reference images is extracted through a convolutional network; the feature information is probabilistically estimated through a uniform pooling layer and a Softmax layer, so that a more accurate probability distribution can be obtained. When the first entropy coded data is arithmetic entropy coded using this probability distribution, it is beneficial to improve coding efficiency and coding accuracy.

[0081] When estimating the above probability distribution through a neural network, introducing the residuals of multiple reference images can obtain a more accurate probability distribution estimate, which is beneficial to improving coding efficiency and coding accuracy.

[0082] In one possible design, the input data includes residuals of multiple reference images, predictions of multiple reference images, and a prediction of a current image, where the multiple reference images are multiple encoded images before encoding the current image; the neural network includes a convolutional network, a uniform pooling layer, and a Softmax layer; the convolutional network includes multiple convolutional layers and ReLU layers, wherein the neural network performs probability estimation processing on the input data to obtain a probability distribution of residual values ​​of multiple samples included in the residual of the current image, including:

[0083] The convolutional network is used to extract the residuals of multiple reference images, the predictions of multiple reference images, and the predicted feature information of the current image; the uniform pooling layer and the Softmax layer are used to perform probability estimation on the feature information extracted by the convolutional network, and the probability distribution of the residual values ​​of multiple samples included in the residual of the current image is output.

[0084] When estimating the above-mentioned probability distribution through a neural network, the residuals and predictions of multiple reference images and the prediction of the current image are introduced, and the feature information of the residuals of multiple reference images, the predictions of multiple reference images and the prediction of the current image is extracted through a convolutional network; the probability of this feature information is estimated through a uniform pooling layer and a Softmax layer, and a more accurate probability distribution can be obtained. When using this probability distribution to perform arithmetic entropy coding on the first entropy coded data, it is beneficial to improve coding efficiency and coding accuracy.

[0085] It should be pointed out here that in this application, except for the residual of the current image to be encoded on the encoding side, which is obtained based on the difference between the original value and the predicted value of the sample of the image, the residuals elsewhere on the encoding side and the residuals appearing on the decoding side all refer to reconstructed residuals.

[0086] Optionally, on the encoding side, the reconstructed residual of the reference image can be obtained by decoding the code stream of the reference image. For the specific decoding process, please refer to the decoding method of the first aspect. The reconstructed residual of the reference image can also be obtained by:

[0087] The residual of the reference image is quantized to obtain the residual of the quantized reference image; the residual of the quantized reference image is then dequantized to obtain the reconstructed residual of the reference image. During the encoding process, after obtaining the residual of the reference image (denoted as R), the residual of the reference image is quantized to obtain the residual of the second image after reference (denoted as R1), and then R1 is entropy encoded to obtain the code stream (denoted as R3). Because entropy coding is lossless, R1 can be directly dequantized to obtain the reconstructed residual of the reference image. This is the same as entropy decoding R3 and then dequantizing the decoded result, which is the same as the reconstructed residual of the reference image mentioned above.

[0088] Based on a third aspect, the present application relates to a video image decoding device. The beneficial effects can be found in the description of the first aspect and are not further elaborated here. The decoding device has the function of implementing the behavior described in the method example of the first aspect. The functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0089] Based on a fourth aspect, the present application relates to a video image encoding device. The beneficial effects can be found in the description of the second aspect and are not further elaborated here. The encoding device has the function of implementing the behavior described in the method example of the second aspect. The functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0090] The method described in the first aspect of the present application can be performed by the apparatus described in the third aspect of the present application. Other features and implementations of the method described in the first aspect of the present application directly depend on the functionality and implementation of the apparatus described in the third aspect of the present application.

[0091] The method described in the second aspect of this application can be performed by the apparatus described in the fourth aspect of this application. Other features and implementations of the method described in the second aspect of this application directly depend on the functionality and implementation of the apparatus described in the fourth aspect of this application.

[0092] Based on a fifth aspect, the present application relates to an apparatus for decoding a video stream, comprising a processor and a memory, wherein the memory stores instructions, and the instructions enable the processor to execute the method described in the first aspect.

[0093] Based on a sixth aspect, the present application relates to an apparatus for encoding a video stream, comprising a processor and a memory, wherein the memory stores instructions, and the instructions enable the processor to execute the method according to the second aspect.

[0094] According to a seventh aspect, a computer-readable storage medium is provided, having stored thereon instructions that, when executed, cause one or more processors to encode video data. The instructions cause the one or more processors to perform the method of the first or second aspect, or any possible embodiment of the first or second aspect.

[0095] Based on the eighth aspect, the present application relates to a computer program product comprising program code, which, when run, executes the method of the first or second aspect or any possible embodiment of the first or second aspect.

[0096] One or more embodiments will be described in detail in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0098] Figure 1 is a block diagram of an example of a video decoding system for implementing an embodiment of the present application;

[0099] Figure 2 is a block diagram of another example of a video decoding system for implementing an embodiment of the present application;

[0100] Figure 3 is a schematic block diagram of a video decoding device for implementing an embodiment of the present application;

[0101] Figure 4 is a schematic block diagram of a video decoding device for implementing an embodiment of the present application;

[0102] Figure 5 Schematic diagram of the comparison of the original image, predicted image and residual;

[0103] Figure 6A A schematic diagram of the structure of a video encoder provided in an embodiment of the present application;

[0104] Figure 6B A schematic diagram illustrating a residual signal acquisition principle according to an embodiment of the present application;

[0105] Figure 6C This is another schematic diagram of the residual signal acquisition principle according to an embodiment of the present application;

[0106] Figure 6DA schematic diagram of the structure of a coding network according to an embodiment of the present application;

[0107] Figure 6E A schematic diagram of another coding network structure illustrated in an embodiment of the present application;

[0108] Figure 6F A schematic diagram of another coding network structure illustrated in an embodiment of the present application;

[0109] Figure 6G A schematic diagram of the structure of the network layer according to an embodiment of the present application;

[0110] Figure 6H Schematic diagram for explaining probability distribution;

[0111] Figure 7A A schematic diagram of the structure of a neural network for probability estimation according to an embodiment of the present application;

[0112] Figure 7B This is a schematic diagram of the structure of another neural network for probability estimation according to an embodiment of the present application;

[0113] Figure 7C A schematic diagram of a hidden information acquisition principle according to an embodiment of the present application;

[0114] Figure 7D This is a schematic diagram illustrating another hidden information acquisition principle according to an embodiment of the present application;

[0115] Figure 7E This is a schematic diagram of arithmetic coding interval division;

[0116] Figure 8A A schematic diagram of the structure of another video encoder provided in an embodiment of the present application;

[0117] Figure 8B A schematic diagram of the structure of another video encoder provided in an embodiment of the present application;

[0118] Figure 9A A schematic diagram of the structure of a video decoder provided in an embodiment of the present application;

[0119] Figure 9B This is a functional diagram of a decoding network according to an embodiment of the present application;

[0120] Figure 9C A schematic diagram of the structure of a decoding network according to an embodiment of the present application;

[0121] Figure 9D A schematic diagram of the structure of another decoding network according to an embodiment of the present application;

[0122] Figure 9EA schematic diagram of the structure of another video decoder according to an embodiment of the present application;

[0123] Figure 9F A schematic diagram of the structure of another video decoder according to an embodiment of the present application;

[0124] Figure 10 1000 is a flowchart illustrating a coding method according to an embodiment of the present application;

[0125] Figure 11A A flowchart illustrating a probability distribution estimation process based on the present application;

[0126] Figure 11B A flowchart illustrating a probability distribution estimation process based on the present application;

[0127] Figure 12 12 is a flowchart illustrating a decoding method according to an embodiment of the present application;

[0128] Figure 12A A flowchart illustrating a probability distribution estimation process based on the present application;

[0129] Figure 12B A flowchart illustrating a probability distribution estimation process based on the present application;

[0130] Figure 13 A schematic diagram of an encoding and decoding process according to an embodiment of the present application;

[0131] Figure 14 This is another schematic diagram of the encoding and decoding process according to an embodiment of the present application;

[0132] Figure 15 This is another schematic diagram of the encoding and decoding process according to an embodiment of the present application;

[0133] Figure 16 This is another schematic diagram of the encoding and decoding process according to an embodiment of the present application;

[0134] Figure 17 This is another schematic diagram of the encoding and decoding process according to an embodiment of the present application;

[0135] Figure 18 This is a schematic diagram of another encoding and decoding process illustrated in an embodiment of the present application. DETAILED DESCRIPTION

[0136] The embodiments of the present application provide an AI-based video image compression technology, in particular, a neural network-based video compression technology, and specifically provide an entropy coding and decoding technology for video images to improve the traditional hybrid video coding and decoding system.

[0137] Video coding generally refers to the processing of a sequence of images to form a video or video sequence. In the field of video coding, the terms "picture", "frame" or "image" can be used as synonyms. Video coding (or commonly referred to as coding) includes two parts: video encoding and video decoding. Video coding is performed on the source side and generally includes processing (for example, compressing) the original video image to reduce the amount of data required to represent the video image (thereby making it more efficient to store and / or transmit). Video decoding is performed on the destination side and generally includes inverse processing relative to the encoder to reconstruct the video image. The "encoding" of the video image (or commonly referred to as image) involved in the embodiment should be understood as the "encoding" or "decoding" of the video image or video sequence. The encoding part and the decoding part are also collectively referred to as codec (encoding and decoding, CODEC).

[0138] In the case of lossless video coding, the original video image can be reconstructed, that is, the reconstructed video image has the same quality as the original video image (assuming there is no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, further compression is performed through quantization, etc. to reduce the amount of data required to represent the video image, but the decoder side cannot fully reconstruct the video image, that is, the quality of the reconstructed video image is lower or worse than the quality of the original video image.

[0139] Since the embodiments of the present application involve the application of a large number of neural networks, for ease of understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0140] (1) Neural Network

[0141] A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0142]

[0143] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal of the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0144] (2) Deep Neural Networks

[0145] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. Based on the location of the different layers, the neural network within a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. Each layer is fully connected, meaning that any neuron in layer i is connected to any neuron in layer i+1.

[0146] Although DNN looks complicated, the work of each layer is actually not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and a() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0147] In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as

[0148] It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0149] (3) Convolutional Neural Networks

[0150] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. It consists of a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each of which can be composed of a rectangular arrangement of neurons. Neurons in the same feature plane share weights, which are referred to as convolution kernels. Shared weights can be understood as extracting image information in a position-independent manner. Convolution kernels can be initialized as matrices of random size, and during CNN training, they can learn to acquire reasonable weights. Furthermore, shared weights have the direct benefit of reducing the number of connections between layers of the CNN, thereby reducing the risk of overfitting.

[0151] (4) Recurrent neural networks (RNNs) are used to process sequence data. In traditional neural network models, the layers are fully connected from the input layer to the hidden layer and then to the output layer, while the nodes within each layer are disconnected. Although this ordinary neural network solves many difficult problems, it is still powerless to solve many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word because the previous and next words in a sentence are not independent. RNN is called a recurrent neural network because the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer disconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN. RNN aims to make machines have the ability to remember like humans. Therefore, the output of RNN needs to rely on the current input information and historical memory information.

[0152] (5) Loss function

[0153] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which is to pre-configure the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss as much as possible.

[0154] (6) Backpropagation algorithm

[0155] Neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during training, reducing the reconstruction error loss of the neural network model. Specifically, the forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial neural network model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0156] In the following embodiment of the decoding system 10, the encoder 20 and the decoder 30 are based on Figures 1 to 2 Provide a description.

[0157] Figure 1 FIG1 is a schematic block diagram of an exemplary decoding system 10, such as a video decoding system 10 (or simply decoding system 10) that can utilize the techniques of the present application. Video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in video decoding system 10 represent devices that can be used to perform various techniques based on the various examples described in this application.

[0158] like Figure 1 As shown, the decoding system 10 includes a source device 12 for providing encoded image data 21 such as an encoded image to a destination device 14 for decoding the encoded image data 21.

[0159] The source device 12 includes an encoder 20 , and optionally, may include an image source 16 , a preprocessor (or preprocessing unit) 18 such as an image preprocessor, and a communication interface (or communication unit) 22 .

[0160] The image source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage that stores any of the above images.

[0161] In order to distinguish the processing performed by the pre-processor (or pre-processing unit) 18 , the image (or image data 17 ) may also be referred to as a raw image (or raw image data) 17 .

[0162] The preprocessor 18 is configured to receive (raw) image data 17 and preprocess the image data 17 to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color grading, or denoising. It will be appreciated that the preprocessor 18 may be an optional component.

[0163] The video encoder (or encoder) 20 is configured to receive the pre-processed image data 19 and provide encoded image data 21 .

[0164] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device through the communication channel 13 for storage or direct reconstruction.

[0165] The destination device 14 includes a decoder 30 and, in addition or alternatively, may include a communication interface (or communication unit) 28 , a post-processor (or post-processing unit) 32 , and a display device 34 .

[0166] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is a encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0167] The communication interface 22 and the communication interface 28 can be used to send or receive encoded image data (or encoded data) 21 through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0168] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0169] The communication interface 28 corresponds to the communication interface 22 , and can be used, for example, to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded image data 21 .

[0170] Both the communication interface 22 and the communication interface 28 can be configured as follows Figure 1 The unidirectional communication interface or the bidirectional communication interface indicated by the arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0171] The video decoder (or decoder) 30 is used to receive the encoded image data 21 and provide decoded image data (or decoded image data) 31 (hereinafter referred to as Figure 9A 、 Figure 9E and Figure 9F etc. for further description).

[0172] The post-processor 32 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data) such as the decoded image to obtain post-processed image data 33 such as the post-processed image. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color grading, cropping, or resampling, or any other processing for generating the decoded image data 31 for display on a display device 34 or the like.

[0173] The display device 34 is configured to receive the post-processed image data 33 and display the image to a user or viewer. The display device 34 may be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen.

[0174] The decoding system 10 further includes a training engine 25 for training a neural network in the encoder 20 or decoder 30 to process the reconstructed residual of an input encoded image or the reconstructed residual of a decoded image to obtain a probability distribution of residual values ​​for a plurality of samples included in the residual of the current image. Optionally, the input data also includes hyper-prior information.

[0175] The training data can be stored in a database (not shown), and the training engine 25 trains a neural network based on the training data. The neural network is used to estimate the probability distribution of residual values ​​of multiple samples included in the residual of the current image. It should be noted that the embodiment of the present application does not limit the source of the training data. For example, the training data can be obtained from the cloud or other places to train the neural network for probability estimation.

[0176] The neural network trained by the training engine 25 can be applied to the decoding system 10, 40, for example, Figure 1 The source device 12 (e.g., encoder 20) or the destination device 14 (e.g., decoder 30) shown in FIG. The training engine 25 can train the neural network in the cloud, and then the decoding system 10 downloads and uses the neural network from the cloud; or the training engine 25 can train the neural network in the cloud and use the neural network to obtain a probability distribution of residual values ​​of multiple samples included in the residual of the current image, and the decoding system 10 directly obtains the probability distribution from the cloud.

[0177] although Figure 1The source device 12 and the destination device 14 are shown as independent devices, but the device embodiments may also include both the source device 12 and the destination device 14 or the functions of both the source device 12 and the destination device 14, that is, include both the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions. In these embodiments, the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0178] Based on the description, Figure 1 It will be apparent to the skilled person that the presence and (exact) division of the different units or functions in the source device 12 and / or destination device 14 shown may vary based on the actual device and application.

[0179] The encoder 20 (eg, video encoder 20) or the decoder 30 (eg, video decoder 30), or both, may be configured to generate a video signal by: Figure 2 The processing circuitry shown may be implemented, for example, by one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video encoding processors, or any combination thereof. The encoder 20 may be implemented by processing circuitry 46 to include reference to Figure 2 The various modules discussed in encoder 20 and / or any other decoder systems or subsystems described herein. Decoder 30 may be implemented by processing circuitry 46 to include reference to Figure 3 The processing circuitry 46 may be used to perform the various operations discussed below. Figure 4 As shown, if part of the technology is implemented in software, the device can store the software instructions in a suitable non-transitory computer-readable storage medium and use one or more processors to execute the instructions in hardware to perform the technology of the present application. One of the video encoder 20 and the video decoder 30 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as Figure 2 shown.

[0180] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook computer or laptop, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or content distribution server), a broadcast receiving device, a broadcast transmitting device, etc., and may not use or use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.

[0181] In some cases, Figure 1 The video decoding system 10 shown is merely exemplary, and the techniques provided herein may be applicable to video encoding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other examples, data is retrieved from a local memory, sent over a network, and so on. A video encoding device may encode data and store the data in a memory, and / or a video decoding device may retrieve data from a memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but only encode data to a memory and / or retrieve and decode data from a memory.

[0182] Figure 2 FIG4 is an illustrative diagram of an example of a video decoding system 40 including a video encoder 20 and / or a video decoder 30 according to an exemplary embodiment. The video decoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video codec implemented by a processing circuit 46), an antenna 42, one or more processors 43, one or more memory storage devices 44, and / or a display device 45.

[0183] like Figure 2 As shown, imaging device 41, antenna 42, processing circuit 46, video encoder 20, video decoder 30, processor 43, memory storage 44, and / or display device 45 can communicate with each other. In different embodiments, video decoding system 40 can include only video encoder 20 or only video decoder 30.

[0184] In some instances, antenna 42 can be used to transmit or receive an encoded bitstream of video data. Additionally, in some instances, display device 45 can be used to present the video data. Processing circuitry 46 can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. Video decoding system 40 can also include an optional processor 43, which can similarly include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. Furthermore, memory storage 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory). In a non-limiting example, memory storage 44 can be implemented as cache memory. In other instances, processing circuitry 46 can include memory (e.g., cache memory, etc.) for implementing an image buffer, etc.

[0185] In some examples, video encoder 20 implemented by logic circuitry may include an image buffer (e.g., implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video encoder 20 implemented by processing circuitry 46 to implement reference Figure 2 Logic circuits may be used to perform the various operations discussed herein.

[0186] In some examples, video decoder 30 may be implemented in a similar manner by processing circuitry 46 to implement reference Figure 2 The various modules discussed herein may include video decoder 30 and / or any other decoder systems or subsystems described herein. In some examples, logic circuitry implemented video decoder 30 may include an image buffer (implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video decoder 30 implemented by processing circuitry 46.

[0187] In some examples, antenna 42 may be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream may include data related to the encoded video frames, indicators, index values, mode selection data, etc., as discussed herein, such as data related to encoding partitions (e.g., transform coefficients or quantized transform coefficients, optional indicators (as discussed), and / or data defining encoding partitions). Video decoding system 40 may also include video decoder 30 coupled to antenna 42 and configured to decode the encoded bitstream. Display device 45 is configured to present the video frames.

[0188] It should be understood that for the examples described herein with reference to video encoder 20, video decoder 30 can be configured to perform the reverse process. With respect to signaling syntax elements, video decoder 30 can be configured to receive and parse such syntax elements and decode the associated video data accordingly. In some examples, video encoder 20 can entropy encode the syntax elements into an encoded video bitstream. In such examples, video decoder 30 can parse such syntax elements and decode the associated video data accordingly.

[0189] It should be noted that the encoding and decoding process described in this application exists in most video codecs, such as H.263, H.264, MPEG-2, MPEG-4, VP8, VP9, ​​AI-based end-to-end image coding and other corresponding codecs.

[0190] Figure 3 Schematic diagram of a video decoding device 400 provided in an embodiment of the present application. The video decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 400 may be a decoder, such as Figure 2 The video decoder 30 in , may also be an encoder, such as Figure 2 The video encoder 20 in.

[0191] Video decoding device 400 includes: an input port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; for example, processor 430 may be a neural network processor 430; a transmitter unit (Tx) 440 and an output port 450 (or output port 450) for transmitting data; and a memory 460 for storing data. Video decoding device 400 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to input port 410, receiver unit 420, transmitter unit 440, and output port 450 for outputting or transmitting optical or electrical signals.

[0192] The processor 430 is implemented in hardware and software. The processor 430 can be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the input port 410, the receiving unit 420, the sending unit 440, the output port 450, and the memory 460. The processor 430 includes a decoding module 470 (e.g., a decoding module 470 based on a neural network NN). The decoding module 470 implements the embodiments disclosed above. For example, the decoding module 470 performs, processes, prepares, or provides various encoding operations. Therefore, the decoding module 470 provides substantial improvements to the functionality of the video decoding device 400 and affects the switching of the video decoding device 400 to different states. Alternatively, the decoding module 470 is implemented by instructions stored in the memory 460 and executed by the processor 430.

[0193] Memory 460 includes one or more disks, tape drives, and solid-state drives and can be used as overflow data storage for storing programs when such programs are selected for execution, and for storing instructions and data read during program execution. Memory 460 can be volatile and / or non-volatile and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0194] Figure 4 A simplified block diagram of an apparatus 500 is provided for an exemplary embodiment. The apparatus 500 may be used as Figure 1 Either or both of the source device 12 and the destination device 14 in .

[0195] The processor 502 in the apparatus 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices, now available or developed in the future, capable of manipulating or processing information. While the disclosed implementations may be implemented using a single processor, such as the processor 502 shown, using more than one processor may provide greater speed and efficiency.

[0196] In one implementation, the memory 504 in the apparatus 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 that are accessed by the processor 502 via a bus 512. The memory 504 may also include an operating system 508 and application programs 510, which include at least one program that allows the processor 502 to perform the methods described herein. For example, the application programs 510 may include applications 1 through N, as well as a video decoding application that performs the methods described herein.

[0197] The apparatus 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display that combines a display with touch-sensitive elements that can be used to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0198] Although bus 512 in device 500 is described herein as a single bus, bus 512 may include multiple buses. Furthermore, secondary storage may be directly coupled to other components of device 500 or accessed via a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, device 500 may have a variety of configurations.

[0199] like Figure 5 As shown in the figure, the first row is the original image, the second row is the prediction of the original image, and the third row is the residual of the original image. As can be seen from the figure, there is a clear correlation between the residuals of consecutive images. Based on this correlation, the residual of the current image can be encoded and the code stream of the current image can be decoded, thereby improving the efficiency of encoding and decoding.

[0200] It should be understood that there is also correlation between the residuals of non-consecutive images, but the difference between the timestamps of non-consecutive images is less than a preset threshold, so that the content in the non-consecutive images is mostly the same, thereby ensuring that there is obvious correlation between the residuals of the non-consecutive images.

[0201] Encoders and encoding methods

[0202] Figure 6A Schematic block diagram of an example of a video encoder for implementing the technology of the present application. Figure 6A In the example of , the video encoder 20 includes an input end (or input interface) 202, an encoding network 204, a rounding unit 206, an entropy encoding unit 208, a neural network 210 for probability estimation, and an output end (or output interface) 212. Figure 6A The illustrated video encoder 20 may also be referred to as an end-to-end video encoder or a video encoder based on an end-to-end video encoder.

[0203] The encoder 20 may receive input data 203 via an input terminal 202 or the like, where the input data image includes a residual of a current image as an input to an encoding network 204 .

[0204] The input data 203 also includes the residual of the reference image, or,

[0205] The residuals of multiple reference images, or

[0206] The residuals of multiple reference images and the prediction of the current image, or,

[0207] The residuals of the multiple reference images, the predictions of the multiple reference images and the prediction of the current image are used as inputs to the neural network 210 for probability estimation.

[0208] A (digital) image is, or can be considered to be, a two-dimensional array or matrix of pixels with intensity values. The pixels in the array are also referred to as pixels (or pels, short for picture elements). The number of pixels in the array or image in the horizontal and vertical directions (or axes) determines the image size and / or resolution. To represent color, three color components are typically used; that is, an image can be represented as or include three pixel arrays. In the RBG format or color space, an image includes corresponding arrays of red, green, and blue pixels. However, in video or image coding, each pixel is typically represented in a luminance / chrominance format or color space, such as YCbCr, which includes a luminance component indicated by Y (sometimes also indicated by L) and two chrominance components, indicated by Cb and Cr. The luminance (luma) component Y represents the brightness or grayscale level intensity (for example, in grayscale images, both are the same), while the two chrominance (chroma) components Cb and Cr represent the chrominance or color information components. Accordingly, an image in YCbCr format includes a luma pixel array of luma pixel values ​​(Y) and two chroma pixel arrays of chroma values ​​(Cb and Cr). An image in RGB format can be converted or transformed into YCbCr format, and vice versa, a process also known as color conversion or transformation. If the image is black and white, the image may include only a luma pixel array. Accordingly, the image may be, for example, a luma pixel array in monochrome format or a luma pixel array and two corresponding chroma pixel arrays in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0209] Among them, the residual of the current image is obtained based on the current image and the prediction of the current image. Specifically, the current image includes the original values ​​of multiple samples (pixel points), and the prediction of the current image includes the predicted values ​​of multiple samples (pixel points). The original values ​​of the samples (pixel points) in the current image are subtracted from the predicted values ​​of the samples at the corresponding positions in the prediction of the current image one by one to obtain the residual of the current image.

[0210] In one example, if Figure 6B As shown, the reconstructed sample values ​​of the coded image are predicted to obtain the prediction signal of the current image; then the original signal of the current image and the prediction of the current image are processed according to the above method to obtain the residual of the current image. In another example, as Figure 6C As shown, the low-resolution image of the current image is upsampled to obtain a prediction of the current image, and then the original signal of the current image and the prediction of the current image are processed in the above manner to obtain a residual of the current image; wherein the low-resolution image of the current image is obtained by downsampling the current image.

[0211] It should be noted that the prediction signal of the current image in this application can also be referred to as the prediction of the current image, and the prediction signal of the reference image can also be referred to as the prediction of the reference image; the original signal of the current image refers to the current image itself. The residual of the current image in this application can also be referred to as the residual signal of the current image.

[0212] Coding Network 204

[0213] like Figure 6A As shown, the encoding network 204 is used to process the residual of the current image through the encoding network to obtain a feature map 205 of the residual of the current image;

[0214] In one possibility, the encoding network 204 is as follows Figure 6D As shown, the encoding network 204 includes K sub-encoding networks, each of which corresponds to the feature layer and the corresponding output feature map. The residual of the current image is input into the first sub-encoding network for feature extraction to obtain the first output feature map; the first output feature map is then input into the second sub-encoding network for feature extraction to obtain the second output feature map, and so on. The K-1th output feature map is input into the Kth sub-encoding network for feature extraction to obtain the Kth output feature map; at this time, there are K output feature maps output, where K>=1. The feature map of the residual of the current image can be any one of the K output feature maps, or the Kth output feature map, or a multi-channel feature map obtained by stacking K output feature maps, and the feature map has multi-scale features.

[0215] In one possibility, the structure of the encoding network 204 is as follows Figure 6E As shown, the encoding network 204 is composed of T network layers, where M, L, T, and K are positive integers. The Mth output feature map and the Kth output feature map are both outputs of the encoding network, and the Kth output feature map output by the encoding network is output after the network layer L of the encoding network 204, and the Mth output feature map is output after the network layer T. It can be understood that there can be multiple output feature maps output after different network layer positions in the encoding network 204, which is not limited here. The feature map of the residual of the above-mentioned current image can be the Mth output feature map, or a multi-channel feature map obtained by stacking multiple output feature maps, and the feature map has multi-scale features.

[0216] It should be pointed out here that before stacking multiple output feature maps, if the sizes of the multiple output feature maps are inconsistent, the multiple output feature maps are processed, such as upsampling, downsampling, etc., so that the sizes of the multiple output feature maps are consistent.

[0217] in Figure 6E Any network layer in can be a convolutional layer, a normalization layer, a nonlinear activation layer, etc.

[0218] In one possibility, the coding network 204 is Figure 6D The corresponding sub-coding network structures can be the same or different from each other. The network structures of the Mth and Nth sub-coding networks are as follows: Figure 6F Wherein the network layer 1 of the Mth sub-coding network and any network layer structure in the Nth sub-coding network can be different from each other, and L and P can be the same or different, and both L and P are positive integers greater than 0.

[0219] In one possibility, the network structure of any sub-coding network in the coding network 204 is as follows: Figure 6G As shown in the figure, the Nth sub-coding network in the example contains 5 network layers, including a normalization layer, two convolutional layers and two non-linear activation layers.

[0220] Rounding unit 206

[0221] The rounding unit 206 is used to round the output feature map 205 by, for example, scalar quantization or vector quantization to obtain an integerized feature map 207.

[0222] In one possibility, the video encoder 20 (correspondingly, the rounding unit 206) may be configured to output a quantization parameter (QP), for example, directly output or output after being encoded or compressed by the entropy coding unit 208, for example, so that the video decoder 30 may receive and use the quantization parameter for decoding.

[0223] Neural Networks for Probability Estimation 210

[0224] The neural network 210 for probability estimation is used to estimate the probability distribution of residual values ​​of multiple samples included in the residual of the current image based on input data.

[0225] Optionally, the above probability distribution represents the probability distribution of multiple difference values ​​between the original values ​​of multiple samples included in the current image and the predicted values ​​of the multiple samples, or represents the probability distribution of the residual values ​​of each sample point in the multiple samples of the current image, or represents the corresponding probability distribution of the residual values ​​of multiple samples of the current image.

[0226] The probability distribution information of the residual values ​​of multiple samples included in the residual of the current image is explained as follows:

[0227] Take a 2x2 size image as an example, Figure 6HAs shown, C0, C1, C2, and C3 are the residuals of the four samples of the current image, and their values ​​are -2, 1, -2, and 0 respectively. R0, R1, R2, and R3 are the residuals of the four samples of the reference image, and their values ​​are -1, 2, -2, and 1 respectively. Assume that the residual value range is [-2, 2], that is, integers between -2 and 2. The residual of the reference image is probabilistically estimated by the neural network 210 for probability estimation, and the probability distribution of the residual values ​​of the four samples included in the residual of the current image is obtained:

[0228] In one case, the probability distribution of the residual values ​​of the four samples included in the residual of the current image specifically refers to the probability distribution of the residual values ​​of each sample point in the four samples of the current image, that is, the probability distribution of the residual value at the C0 position, the probability distribution of the residual value at the C1 position, the probability distribution of the residual value at the C2 position, and the probability distribution of the residual value at the C3 position are shown in the following Table 1:

[0229] Location The probability of a value of -2 The probability of a value of -1 The probability of a value of 0 The probability of a value of 1 The probability of a value of 2 C0 0.6 0.1 0.1 0.1 0.1 C1 0.1 0.1 0.1 0.5 0.2 C2 0.5 0.2 0.1 0.1 0.1 C3 0.1 0.2 0.4 0.2 0.1

[0230] Table 1

[0231] The above-mentioned residual can be subjected to arithmetic entropy coding according to the above-mentioned probability distribution.

[0232] Another possible scenario is to obtain the probability distribution of the residual values ​​of the four samples in the current image based on the residual values ​​of the four samples R0, R1, R2, and R3 in the reference image. For example, based on the residual values ​​of the four samples R0, R1, R2, and R3, which are -1, 2, -2, and 1 respectively, the probability distribution of the residual values ​​of the four samples in the current image is obtained, as shown in Table 2 below.

[0233]

[0234] Table 2

[0235] Or as shown in Table 3 below:

[0236] Location The probability of a value of -2 The probability of a value of -1 The probability of a value of 0 The probability of a value of 1 The probability of a value of 2 C0 0.5 0.2 0.1 0.1 0.1 C1 0.5 0.2 0.1 0.1 0.1 C2 0.5 0.2 0.1 0.1 0.1 C3 0.5 0.2 0.1 0.1 0.1

[0237] Table 3

[0238] Compared with Table 1, the probability distributions shown in Table 2 and Table 3 are the probability distributions of each value within the range of the residual value, and do not distinguish the location of the sample. That is, the probability distributions of the residual values ​​of multiple samples are the same.

[0239] The above-mentioned residual may be encoded by performing arithmetic entropy coding on the above-mentioned residual according to the above-mentioned probability distribution.

[0240] Optionally, the input data of the neural network 210 for probability estimation includes the residual of the reference image, or,

[0241] The residual of the reference image, the prediction of the current image and the prediction of the reference image, or,

[0242] The residuals of multiple reference images, or

[0243] Residuals from multiple reference images and predictions from multiple images.

[0244] It should be noted that the above-mentioned parameter image or multiple reference images are encoded images before the current image is encoded.

[0245] In one example, if Figure 7A As shown, the neural network 210 for probability estimation includes a convolutional network, a uniform pooling layer, and a normalized exponential function Softmax layer, wherein the convolutional network includes an excitation ReLU layer and multiple convolutional layers.

[0246] When the input data of the neural network 210 for probability estimation includes the residual of the reference image, the feature information of the residual of the reference image is extracted through the convolutional network; when the input data of the neural network 210 for probability estimation includes the residual of the reference image, the prediction of the reference image and the prediction of the current image, the feature information of the residual of the reference image, the prediction of the reference image and the prediction of the current image is extracted through the convolutional network; when the input data of the neural network 210 for probability estimation includes the residual of multiple reference images, the feature information of the residual of multiple reference images is extracted through the convolutional network; when the input data of the neural network 210 for probability estimation includes the residual of multiple reference images, the prediction of multiple reference images and the prediction of the current image, the feature information of the residual of multiple reference images, the prediction of multiple reference images and the prediction of the current image is extracted through the convolutional network; then the feature information extracted by the convolutional network is probability estimated through the uniform pooling layer and the Softmax layer, and the probability distribution of the residual values ​​of multiple samples included in the residual of the current image is output.

[0247] Optionally, when the input of the neural network 210 for probability estimation includes multiple images (such as the residual of a reference image, the prediction of a reference image and the prediction of a current image, or the residuals of multiple reference images, or the residuals of multiple reference images, the predictions of multiple reference images and the prediction of the current image), before the multiple images are input into the neural network 210 for probability estimation, the multiple images are first stacked to obtain a multi-channel image, and then the multi-channel image is input into the neural network 210 for probability estimation. The convolutional network in the neural network 210 for probability estimation extracts feature information of the multi-channel image, and then probability estimation is performed on the feature information extracted by the convolutional network through a uniform pooling layer and a Softmax layer, and the probability distribution of the residual values ​​of the multiple samples included in the residual of the current image is output.

[0248] In one example, the input data of the neural network 210 for probability estimation also includes hyper-prior information. In this case, the structure of the neural network 210 for probability estimation is the same as Figure 7A are different, such as Figure 7B As shown in the figure, the neural network includes a convolutional network, a convolutional pooling layer, a first convolutional layer, a concatenation layer, and a Softmax layer. The convolutional network includes multiple convolutional layers and ReLU layers.

[0249] When the input data of the neural network 210 for probability estimation includes the residual and super-prior information of the reference image, the feature information of the residual and super-prior information of the reference image is extracted through the convolutional network; when the input data of the neural network 210 for probability estimation includes the residual of the reference image, the prediction of the reference image, the super-prior information and the prediction of the current image, the feature information of the residual of the reference image, the prediction of the reference image, the super-prior information and the prediction of the current image is extracted through the convolutional network; when the input data of the neural network 210 for probability estimation includes the residual and super-prior information of multiple reference images, the feature information of the residual and super-prior information of the reference image is extracted through the convolutional network The network extracts feature information of residuals and super-prior information of multiple reference images; when the input data of the neural network 210 for probability estimation includes residuals of multiple reference images, predictions of multiple reference images, super-prior information and prediction of the current image, the feature information of the residuals of multiple reference images, predictions of multiple reference images, super-prior information and prediction of the current image is extracted through the convolutional network; then the feature information extracted by the convolutional network is probability estimated through the splicing layer, the first convolutional layer, the uniform pooling layer and the Softmax layer, and the probability distribution of the residual values ​​of multiple samples included in the residual of the current image is output.

[0250] Optionally, the first convolutional layer may be the same as or different from the convolutional layers in the convolutional network.

[0251] In one example, when the input data of the neural network 210 for probability estimation includes the residual of the reference image, the prediction of the current image, and the prediction of the reference image, before the residual of the reference image, the prediction of the current image, and the prediction of the reference image are input into the neural network 210 for probability estimation, a stacking operation is performed on the residual of the reference image, the prediction of the current image, and the prediction of the reference image to obtain a three-channel image, which is then input into the neural network 210 for probability estimation for processing; or,

[0252] When the input data of the neural network 210 for probability estimation includes residuals of multiple reference images, before the residuals of the multiple reference images are input into the neural network 210 for probability estimation, the residuals of the multiple reference images are stacked to obtain a multi-channel image, and the multi-channel image is then input into the neural network 210 for probability estimation for processing, or,

[0253] When the input data of the neural network 210 for probability estimation includes residuals of multiple reference images, predictions of multiple images, and predictions of the current image, before being input into the neural network 210 for probability estimation, the residuals of the multiple reference images, the predictions of the multiple images, and the predictions of the current image are stacked to obtain a multi-channel image, and then the multi-channel image is input into the neural network 210 for probability estimation for processing.

[0254] In another example, when the input data of the neural network 210 for probability estimation includes residuals of multiple reference images, before the residuals of the multiple reference images are input into the neural network 210 for probability estimation, the residuals of the multiple reference images are passed through a recurrent neural network to obtain first hidden information, and then the first hidden information is input into the neural network 210 for probability estimation for processing.

[0255] Specifically, the residual P of the jth reference image among the residuals of multiple reference images is j and hidden information H j-1 Input into the recurrent neural network for processing to obtain the hidden information H j ; The residuals of multiple reference images are processed in this way to obtain the above-mentioned first hidden information; wherein, when the residual of the j-th reference image is the first of the residuals of multiple reference images, the hidden information H j-1 When the residual of the jth reference image is the last one among the residuals of multiple reference images, the hidden information H j The first hidden information.

[0256] The difference between the time stamps of two adjacent reference images in the plurality of reference images is the preset threshold.

[0257] For example, Figure 7C As shown, it is assumed that the residuals of N reference code images are subjected to a recurrent neural network to obtain the first hidden information, which specifically includes:

[0258] First, the residual of the first reference image is input into the recurrent neural network for processing to obtain the hidden information H1; then the hidden information H1 and the residual of the second reference image are input into the recurrent neural network for processing to obtain the hidden information H2; ...the hidden information H N-3 The residual of the N-2th reference image is input into the recurrent neural network for processing to obtain the hidden information H N-2 ; Hide the information H N-2 The residual of the N-1th reference image is input into the recurrent neural network for processing to obtain the hidden information H N-1 ; Hide the information H N-1The residual of the Nth reference image is input into the recurrent neural network for processing to obtain the hidden information H N ; The hidden information H N The first hidden information.

[0259] In another example, when the input data of the neural network 210 for probability estimation includes residuals of multiple reference images, predictions of multiple reference images, and a prediction of the current image, before being input into the neural network 210 for probability estimation, the residuals of the multiple reference images and the predictions of the multiple reference images are processed through a recurrent neural network to obtain second hidden information, and the second hidden information and the prediction of the current image are then input into the neural network 210 for probability estimation for processing. The second hidden information is essentially a feature map, so after obtaining the second hidden information, the second hidden information and the prediction of the current image can be stacked to obtain a two-channel image, which is then input into the neural network 210 for processing.

[0260] Optionally, obtaining the second hidden information based on the residuals of the multiple reference images and the predictions of the multiple reference images includes:

[0261] The residual P of the jth reference image among the residuals of multiple reference images is j , the prediction P of the jth reference image among the predictions of multiple reference images j ' and hidden information H j-1 'Input into the recurrent neural network for processing to obtain hidden information H j '; The second hidden information is obtained by processing the residuals of multiple reference images and the predictions of multiple reference images in this way; wherein, when the residual of the j-th reference image is the first of the residuals of the multiple reference images and the prediction of the j-th reference image is the first of the predictions of the multiple reference images, the hidden information H j-1 ' is 0; when the residual of the jth reference image is the last one among the residuals of multiple reference images, and the prediction of the jth reference image is the last one among the predictions of multiple reference images, the hidden information H j ' is the second hidden information.

[0262] For example, Figure 7D As shown, it is assumed that the second hidden information is obtained by the residuals of N reference images and the predictions of N reference images, specifically including:

[0263] First, the residual of the first reference image and the prediction of the first reference image are input into the recurrent neural network for processing to obtain the hidden information H1'; then the hidden information H1', ​​the residual of the second reference image and the prediction of the second reference image are input into the recurrent neural network for processing to obtain the hidden information H2'; ...the hidden information H N-3 ', the residual difference of the N-2th reference image and the prediction of the N-2th reference image are input into the recurrent neural network for processing to obtain the hidden information H N-2 ';Hide the information H N-2 ', the residual of the N-1th reference image and the prediction of the N-1th reference image are input into the recurrent neural network for processing to obtain the hidden information H N-1 ';Hide the information H N-1 , the residual of the Nth reference image and the prediction of the Nth reference image are input into the recurrent neural network for processing to obtain the hidden information H N ';The hidden information H N ' is the second hidden information.

[0264] It should be pointed out here that Figure 7A and Figure 7B In the neural network structure shown, the execution order between layers is not limited to Figure 7A and Figure 7B The execution order shown is, of course, also possible.

[0265] Optionally, the video encoder 20 further includes a super-prior encoding network 218 and a super-prior network decoding network 220, wherein the super-prior encoding network 218 further extracts features from the feature map of the residual of the current image output by the encoding network 204 to obtain a feature vector of the super-prior information, and then inputs the feature vector of the super-prior information into the rounding unit 206, which rounds the feature vector of the super-prior information to obtain an integerized feature vector of the super-prior information; and then inputs the integerized feature vector of the super-prior information into the super-prior decoding network 220 for processing to obtain the super-prior information.

[0266] Optionally, the network architecture of the super-a priori encoding network 218 may refer to the network architecture of the encoding network 204, or may be another form of architecture, which will not be described in detail here. The functions of the super-a priori encoding network 218 are opposite to those of the super-a priori decoding network 220. Therefore, the network structure of the super-a priori decoding network 218 may refer to the network structure of the super-a priori encoding network 218.

[0267] Since the super-prior information is optional for the neural network 210 used for probability estimation, and therefore for the video encoder 20, the super-prior encoding network 218 and the super-prior information decoding network 220 are also optional, the super-prior encoding network 218 and the super-prior information decoding network 220 are represented by dotted boxes.

[0268] Entropy coding unit 208

[0269] The entropy coding unit 208 is configured to apply an entropy coding algorithm or scheme (e.g., a variable length coding (VLC) scheme, a context adaptive VLC (CALVC) scheme, an arithmetic coding scheme, a binarization algorithm, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to the integerized feature map and the probability distribution of residual values ​​of the plurality of samples included in the residual of the current image, thereby obtaining coded image data 21 that can be outputted via an output terminal 212 in the form of a coded bitstream 21, etc., so that a video decoder 30, etc. can receive and use parameters for decoding. The coded bitstream 21 can be transmitted to the video decoder 30 or stored in a memory for later transmission or retrieval by the video decoder 30.

[0270] In one example, since entropy coding does not know the actual character probability of the integerized feature map, it is necessary to count these or related information and add it to the entropy coding unit 208, and the information can be transmitted to the decoding end.

[0271] In one example, since the neural network 210 for probability estimation on the encoding side utilizes super-prior information when estimating the probability distribution of residual values ​​of multiple samples included in the residual of the current image; therefore, after obtaining the super-prior information, the super-prior information is entropy encoded to obtain a super-prior information code stream, and the code stream is transmitted to the video decoder 30 for decoding, so that the video decoder 30 obtains the super-prior information and uses the super-prior information to estimate the above-mentioned probability distribution.

[0272] The following describes the process of arithmetic entropy coding:

[0273] Still Figure 6HFor example, suppose it is known that the residual values ​​of the four samples C0, C1, C2, and C3 need to be arithmetically entropy coded, and their values ​​are -2, 1, -2, and 0 respectively. The probability distribution is shown in Table 1. According to the arithmetic coding algorithm, based on the probability distribution of the residual values ​​of the samples C0, C1, C2, and C3, the interval from 0 to 1 is divided, and the residual values ​​of the samples C0, C1, C2, and C3 are expressed as an interval between real numbers 0 and 1. The lower boundary of the "current interval" output is the encoded codeword. The specific process is as follows: Figure 7E As shown, 0.285 is the final encoded codeword. In actual encoding, this value can be expressed as a fixed point and written into the code stream.

[0274] Similarly, when the probability distribution is as shown in Table 2 or Table 3, arithmetic coding is performed according to the above-mentioned similar method.

[0275] Figure 8A FIG. 1 is a schematic block diagram of an example of another video encoder for implementing the technology of the present application. Figure 8A In the example of , the video encoder 20 includes an input end (or input interface) 202 , a quantization unit 216 , an entropy coding unit 208 , a neural network 210 for probability estimation, and an output end (or output interface) 212 . Figure 8A The illustrated video encoder 20 may also be referred to as an end-to-end video encoder or a video encoder based on an end-to-end video encoder.

[0276] The encoder 20 may receive input data 203 via an input terminal 202 or the like, where the input data image includes a residual of a current image as an input to a quantization unit 216 .

[0277] The input data 203 also includes the residual of the reference image, or,

[0278] The residuals of multiple reference images, or

[0279] The residuals of multiple reference images and the prediction of the current image, or,

[0280] The residuals of the multiple reference images, the predictions of the multiple reference images and the prediction of the current image are used as inputs to the neural network 210 for probability estimation.

[0281] Quantization unit 216

[0282] The quantization unit 216 is used to quantize the residual of the current image to obtain the quantized residual of the current image. The quantized residual of the current image serves as the input of the entropy coding unit 208 .

[0283] Optionally, the quantization may be uniform quantization or non-uniform quantization.

[0284] Neural Networks for Probability Estimation 210

[0285] The detailed functions of the neural network 210 for probability estimation can be found in Figure 6A The description of the neural network 210 for probability estimation in Figure 8A The function of the neural network 210 for probability estimation in Figure 6A The function of the neural network 210 used for probability estimation is the same as that of FIG. 1 and will not be described again here.

[0286] Entropy coding unit 208

[0287] The entropy coding unit 208 is configured to apply an entropy coding algorithm or scheme (e.g., a VLC scheme, CALVC, an arithmetic coding scheme, a binarization algorithm, CABAC, SBAC, PIPE coding, or other entropy coding methods or techniques) to the probability distribution of residual values ​​of a plurality of samples included in the quantized feature map and the residual of the current image, thereby obtaining coded image data 21 that can be outputted in the form of a coded bitstream 21 through an output terminal 212, so that a video decoder 30, etc. can receive and use parameters for decoding. The coded bitstream 21 can be transmitted to the video decoder 30, or stored in a memory for later transmission or retrieval by the video decoder 30.

[0288] It should be noted that the detailed functions of the entropy coding unit 208 can be found in Figure 6A The description of the entropy coding unit 208, or Figure 8A The function of the entropy coding unit 208 in Figure 6A The functions of the medium entropy coding unit 208 are the same and will not be described again here.

[0289] Figure 8B FIG. 1 is a schematic block diagram of an example of another video encoder for implementing the technology of the present application. Figure 8B In the example of , the video encoder 20 includes an input end (or input interface) 202, a transform unit 214, a quantization unit 216, an entropy coding unit 208, a neural network 210 for probability estimation, and an output end (or output interface) 212. Figure 8B The illustrated video encoder 20 may also be referred to as an end-to-end video encoder or a video encoder based on an end-to-end video encoder.

[0290] The encoder 20 may receive input data 203 via an input terminal 202 or the like, where the input data image includes a residual of a current image as input to a transform unit 214 .

[0291] The input data 203 also includes the residual of the reference image, or,

[0292] The residuals of multiple reference images, or

[0293] The residuals of multiple reference images and the prediction of the current image, or,

[0294] The residuals of the multiple reference images, the predictions of the multiple reference images and the prediction of the current image are used as inputs to the neural network 210 for probability estimation.

[0295] Transformation unit 214

[0296] The transform unit 214 is configured to transform the residual of the current image to obtain the transformed residual of the current image, which is used as the input of the quantization unit 216 .

[0297] Optionally, the transformation may be discrete cosine transform (DCT), discrete sine transform (DST), or discrete wavelet transform (DWT).

[0298] Quantization unit 216

[0299] The quantization unit 216 is configured to quantize the residual of the transformed current image to obtain the residual of the transformed and quantized current image, which is used as the input of the entropy coding unit 208 .

[0300] Optionally, the quantization may be uniform quantization or non-uniform quantization.

[0301] Neural Networks for Probability Estimation 210

[0302] The detailed functions of the neural network 210 for probability estimation can be found in Figure 6A The description of the neural network 210 for probability estimation in Figure 8A The function of the neural network 210 for probability estimation in Figure 6A The function of the neural network 210 used for probability estimation is the same as that of FIG. 1 and will not be described again here.

[0303] Entropy coding unit 208

[0304] The entropy coding unit 208 is configured to apply an entropy coding algorithm or scheme (e.g., a VLC scheme, CALVC, an arithmetic coding scheme, a binarization algorithm, CABAC, SBAC, PIPE coding, or other entropy coding methods or techniques) to the probability distribution of residual values ​​of a plurality of samples of the transformed and quantized feature map and the residual of the current image, thereby obtaining coded image data 21 that can be outputted in the form of a coded bitstream 21 through an output terminal 212, so that a video decoder 30, etc. can receive and use parameters for decoding. The coded bitstream 21 can be transmitted to the video decoder 30, or stored in a memory for later transmission or retrieval by the video decoder 30.

[0305] It should be noted that the detailed functions of the entropy coding unit 208 can be found in Figure 6A The description of the entropy coding unit 208, or Figure 8A The function of the entropy coding unit 208 in Figure 6A The functions of the medium entropy coding unit 208 are the same and will not be described again here.

[0306] It should be pointed out here that, on the encoding side, in addition to Figure 6A The coding network 204 in Figure 8A The quantization unit 216 and Figure 8B Except for the residual of the current image input to the transform unit 214 in the above, which is obtained by subtracting the original values ​​of multiple samples of the current image from the predicted values ​​of the multiple samples, the residuals mentioned elsewhere (including the residual of the current image and the residual of the reference image) are all reconstructed residuals. The reconstructed residuals of an image refer to the multiple differences between the reconstructed values ​​of multiple samples of the image and the predicted values ​​of the multiple samples.

[0307] Decoder and decoding method

[0308] Figure 9A Schematic block diagram of an example of a video decoder for implementing the technology of the present application. Figure 9A In the example of , the video decoder 30 includes an input end (or input interface) 302, a neural network 304 for probability estimation, an entropy decoding unit 306, a decoding network 308 and an output end (or output interface) 310. Figure 9A The illustrated video decoder 30 may also be referred to as an end-to-end video decoder or a video decoder based on an end-to-end video decoder.

[0309] The decoder 30 may receive input data through an input terminal 302 or the like, where the input data includes a code stream of a current image as an input to an entropy decoding unit 306 .

[0310] The above-mentioned input data also includes the residual of the reference image, or,

[0311] The residuals of multiple reference images, or

[0312] The residuals of multiple reference images and the prediction of the current image, or,

[0313] The residuals of the multiple reference images, the predictions of the multiple reference images, and the prediction of the current image are used as inputs to the neural network 304 for probability estimation.

[0314] The residual of the current image is obtained based on the reconstructed value of the current image and the prediction of the current image. Specifically, the reconstruction of the current image includes the reconstructed values ​​of multiple samples (pixels), and the prediction of the current image includes the predicted values ​​of multiple samples (pixels). The reconstructed values ​​of the samples (pixels) in the current image are subtracted one by one from the predicted values ​​of the samples at corresponding positions in the prediction of the current image to obtain the residual of the current image. Therefore, after obtaining the residual of the current image, the residual values ​​of the multiple samples (pixels) included in the residual of the current image can be added one by one to the predicted values ​​of the multiple samples (pixels) included in the prediction of the current image to obtain the reconstructed value of the current image.

[0315] In one example, the prediction of the current image is obtained by predicting the reconstructed sample values ​​of the reference image, or by upsampling the reconstructed sample values ​​of a low-resolution image of the current image to obtain a sampled image, the resolution of the sampled image is the same as the resolution of the current image, and the sampled image is the prediction of the current image, wherein the low-resolution image of the current image is obtained by downsampling the current image.

[0316] Neural Networks for Probability Estimation 304

[0317] The neural network 304 for probability estimation is used to estimate the probability distribution of residual values ​​of multiple samples included in the residual of the current image based on input data.

[0318] It should be noted that the probability distribution of the residual values ​​of multiple samples included in the residual of the current image can be explained in Figure 6A The relevant descriptions in the illustrated embodiments will not be repeated here.

[0319] In one example, in order to improve decoding accuracy, the input data of the neural network 304 for probability estimation further includes super-prior information, where the super-prior information is used to characterize features of the residual of the current image at a deeper level.

[0320] It should be noted here that the function of the neural network 304 for probability estimation is consistent with the function of the neural network 210 for probability estimation on the encoding side. The description of the function of the neural network 304 for probability estimation can be found in the relevant description of the neural network 210 for probability estimation on the encoding side, which will not be described again here.

[0321] Optionally, the video decoder 30 further includes a super-a priori decoding network 312, which processes the feature vectors of the integerized super-a priori information obtained by the entropy decoding unit 306 performing arithmetic entropy decoding on the super-a priori bitstream to obtain the super-a priori information. The functions of the super-a priori encoding network 218 and the super-a priori decoding network 312 are opposite, and therefore the network structure of the super-a priori decoding network 312 can be referenced to the network structure of the super-a priori encoding network 218.

[0322] Since the super-prior information is optional for the neural network 304 for probability estimation, and thus the super-prior information decoding network 312 is also optional for the video decoder 30, the super-prior information decoding network 312 is represented by a dotted box.

[0323] Entropy decoding unit 306

[0324] The entropy decoding unit 306 is used to apply an entropy decoding algorithm or scheme to the code stream of the current image and the probability distribution of residual values ​​of multiple samples included in the residual of the current image to obtain a feature map of the residual of the current image.

[0325] The entropy decoding unit 306 may employ a decoding algorithm or scheme corresponding to the encoding scheme of the entropy encoding unit 208 of the encoder 20 .

[0326] Decoding Network 308

[0327] The feature map of the residual of the current image is obtained through the decoding network 308 and the residual of the current image can be output through the output terminal 310. The subsequent video decoder 30 can obtain the reconstructed sample value of the current image through the residual of the current image, that is, the reconstruction of the current image.

[0328] Optionally, the feature map of the residual of the current image is a multi-channel feature map, and the multi-channel feature map includes N decoding feature maps, where N is greater than 1. Figure 9B FIG. 3 is a schematic diagram showing N decoded feature maps input into the decoding network 308 .

[0329] In one possibility, the input and output structure in the decoding network 308 is as follows Figure 9C As shown, the decoding network 308 is composed of P network layers, where P>L and M, L, T, and P are all positive integers. The Tth decoding feature map is input after the network layer L of the decoding network 308. The Tth decoding feature map and the Mth decoding feature map are both inputs to the decoding network. It is understood that multiple decoding feature maps can be input after different network layer positions in the decoding network 308, which is not limited here.

[0330] Figure 9CAny network layer in can be a convolutional layer, an anti-normalization layer, a nonlinear activation layer, etc.

[0331] In one possibility, there may be operations including addition, subtraction, etc. in the decoding network 308.

[0332] In one possibility, the network layer structures in the decoding network 308 can be the same as or different from each other.

[0333] In one possibility, the structure of the decoding network 308 is as follows: Figure 9D As shown in the figure, the decoding network in this example includes 12 network layers, specifically one denormalization layer, six convolutional layers, and four nonlinear activation layers. m, n, and t are all positive integers, and the output of the decoding network 308 is the residual of the current image. The t-th, n-th, and m-th decoding feature maps are all inputs to the decoding network, where t, n, and m are all integers.

[0334] Figure 9E Schematic block diagram of an example of a video decoder for implementing the technology of the present application. Figure 9E In the example of , the video decoder 30 includes an input end (or input interface) 302, a neural network 304 for probability estimation, an entropy decoding unit 306, an inverse quantization unit 314 and an output end (or output interface) 310. Figure 9E The illustrated video decoder 30 may also be referred to as an end-to-end video decoder or a video decoder based on an end-to-end video decoder.

[0335] The decoder 30 may receive input data through an input terminal 302 or the like, where the input data image includes a code stream of a current image as an input to an entropy decoding unit 306 .

[0336] The above input data also includes the residual of the reference image, or,

[0337] The residuals of multiple reference images, or

[0338] The residuals of multiple reference images and the prediction of the current image, or,

[0339] The residuals of the multiple reference images, the predictions of the multiple reference images, and the prediction of the current image are used as inputs to the neural network 304 for probability estimation.

[0340] Neural Networks for Probability Estimation 304

[0341] The detailed function of the neural network 304 for probability estimation can be found in Figure 6A The description of the neural network 210 for probability estimation in Figure 9B The function of the neural network 304 for probability estimation in Figure 6AThe function of the neural network 210 used for probability estimation is the same as that of FIG. 1 and will not be described again here.

[0342] Entropy decoding unit 306

[0343] The entropy decoding unit 306 is configured to apply an entropy decoding algorithm or scheme to the code stream of the current image and the probability distribution of residual values ​​of multiple samples included in the residual of the current image, to obtain a quantized residual of the current image.

[0344] The entropy decoding unit 306 may employ a decoding algorithm or scheme corresponding to the encoding scheme of the entropy encoding unit 208 of the encoder 20 .

[0345] Dequantization unit 314

[0346] The inverse quantization unit 314 is used to inverse quantize the residual of the quantized current image to obtain the residual of the current image that can be output through the output end 310. The subsequent video decoder 30 can obtain the reconstructed sample value of the current image through the residual of the current image, that is, the reconstruction of the current image.

[0347] Optionally, the quantization may be uniform quantization or non-uniform quantization.

[0348] Figure 9F Schematic block diagram of an example of a video decoder for implementing the technology of the present application. Figure 9F In the example of , the video decoder 30 includes an input end (or input interface) 302, a neural network 304 for probability estimation, an entropy decoding unit 306, an inverse quantization unit 314, an inverse transform unit 316 and an output end (or output interface) 310. Figure 9F The illustrated video decoder 30 may also be referred to as an end-to-end video decoder or a video decoder based on an end-to-end video decoder.

[0349] The decoder 30 may receive input data through an input terminal 302 or the like, where the input data image includes a code stream of a current image as an input to an entropy decoding unit 306 .

[0350] The above input data also includes the residual of the reference image, or,

[0351] The residuals of multiple reference images, or

[0352] The residuals of multiple reference images and the prediction of the current image, or,

[0353] The residuals of the multiple reference images, the predictions of the multiple reference images, and the prediction of the current image are used as inputs to the neural network 304 for probability estimation.

[0354] Neural Networks for Probability Estimation 304

[0355] The detailed function of the neural network 304 for probability estimation can be found in Figure 6A The description of the neural network 210 for probability estimation in Figure 9B The function of the neural network 304 for probability estimation in Figure 6A The function of the neural network 210 used for probability estimation is the same as that of FIG. 1 and will not be described again here.

[0356] Entropy decoding unit 306

[0357] The entropy decoding unit 306 is used to apply an entropy decoding algorithm or scheme to the code stream of the current image and the probability distribution of residual values ​​of multiple samples included in the residual of the current image to obtain the residual of the current image after transformation and quantization.

[0358] The entropy decoding unit 306 may employ a decoding algorithm or scheme corresponding to the encoding scheme of the entropy encoding unit 208 of the encoder 20 .

[0359] Dequantization unit 314

[0360] The inverse quantization unit 314 is used to inverse quantize the residual of the current image after transformation and quantization, the residual of the current image after transformation.

[0361] Optionally, the quantization may be uniform quantization or non-uniform quantization.

[0362] Inverse transform unit 316

[0363] The inverse transform unit 316 is used to perform an inverse transform on the residual of the transformed current image to obtain the residual of the current image which can be output through the output terminal 310. The subsequent video decoder 30 can obtain the reconstructed sample value of the current image, i.e., the reconstruction of the current image, through the residual of the current image.

[0364] The above transformation may be DCT, DST, or DWT.

[0365] It should be understood that the processing result of the current step can be further processed in the encoder 20 and the decoder 30 and then output to the next step. For example, after the encoder 20 and the decoder 30, the processing result of the encoder 20 and the decoder 30 can be further operated or processed, such as clipping or shifting operations or filtering.

[0366] It should be noted that the residuals mentioned at the decoding side are all reconstruction residuals, which are the differences between the reconstructed values ​​of multiple samples of the image and the predicted values ​​of the multiple samples. The reconstruction residuals can also be called reconstruction residual signals.

[0367] Figure 101 is a flow chart illustrating a process 1000 of an encoding method according to an embodiment of the present application. The process 1000 may be performed by the video encoder 20. The process 1000 is described as a series of steps or operations. It should be understood that the process 1000 may be performed in various orders and / or may occur simultaneously, not limited to the process 1000. Figure 10 The execution order shown.

[0368] like Figure 10 As shown, the encoding method includes:

[0369] S1001. Obtain the residual of the current image.

[0370] The current image includes multiple samples, and the residual of the current image is the difference between the original values ​​and the predicted values ​​of the multiple samples included in the current image.

[0371] Specifically, obtaining the residual of the current image includes: subtracting the original values ​​of multiple samples of the current image from the predicted values ​​of multiple samples of the current image one by one to obtain the residual values ​​of multiple samples of the current image, and the residual of the current image includes the residual values ​​of the multiple samples, wherein the resolution of the current image is the same as the predicted resolution of the current image.

[0372] It should be noted that the prediction of the current image includes the predicted values ​​of multiple samples of the current image. The process of determining the prediction of the current image can be found in Figure 6B and Figure 6C The relevant description will not be given here.

[0373] Optionally, the current image and the reference image may be consecutive frames or non-consecutive frames, for example, where the difference between the timestamp of the current image and the timestamp of the encoded image is a preset threshold. A low-resolution image of the current image may be obtained by downsampling the current image. The timestamp of the image may be the time when the image was captured.

[0374] S1002. Probability estimation processing is performed on input data through a neural network to obtain a probability distribution of residual values ​​of multiple samples included in the residual of the current image, where the input data at least includes the residual of a reference image, which is an image that has been encoded before encoding the current image.

[0375] In one possible design, the neural network is a trained neural network, and the method of this implementation further includes:

[0376] Acquire training input data, where the training input data includes an encoded image before encoding a sample image, where the sample image is a known encoded image; perform probability estimation processing on the training input data through a neural network to obtain an estimated probability distribution of residual values ​​of multiple samples included in the residual of the sample image; obtain a loss value based on the known probability distribution and the estimated probability distribution of the residual values ​​of multiple samples included in the residual of the sample image; when the loss value converges to a first threshold, or the current number of training times of the neural network is greater than or equal to a second threshold, determine that the current neural network is the neural network corresponding to when the neural network training is completed.

[0377] The input data for training includes:

[0378] The residual of the first reference image includes residual values ​​of a plurality of samples, or;

[0379] residual values ​​of a plurality of samples included in the residual of the first reference image and residual values ​​of a plurality of samples included in the residual of the second reference image, or;

[0380] prediction values ​​of a plurality of samples included in the prediction of the sample image, residual values ​​of a plurality of samples included in the residual of the first reference image, and prediction values ​​of a plurality of samples included in the prediction of the first reference image; or;

[0381] Prediction values ​​of multiple samples included in the prediction of the sample image, residual values ​​of multiple samples included in the residual of the first reference image, prediction values ​​of multiple samples included in the prediction of the first reference image, prediction values ​​of multiple samples included in the prediction of the second reference image, and residual values ​​of multiple samples included in the residual of the second reference image;

[0382] The first reference image and the second reference image are encoded images before the sample image is encoded.

[0383] Optionally, the above probability distribution represents the probability distribution of multiple difference values ​​between the original values ​​of multiple samples included in the current image and the predicted values ​​of the multiple samples, or represents the probability distribution of the residual values ​​of each sample point in the multiple samples of the current image, or represents the corresponding probability distribution of the residual values ​​of multiple samples of the current image.

[0384] In one possible design, the first entropy coded data includes the residual of the current image, or a feature map of the residual of the current image, or the residual of the current image after transformation and quantization.

[0385] Optionally, when the first entropy coded data is a feature map of the residual of the current image, the method of this embodiment further includes: performing feature extraction on the residual of the current image through an encoding network to obtain a feature map of the residual of the current image.

[0386] It should be pointed out here that the specific structure and function of the above coding network can be found in Figure 6D 、 Figure 6E 、 Figure 6F and Figure 6G The relevant description will not be given here.

[0387] Optionally, when the first entropy coded data is a residual of the current image after transformation and quantization, the method of this embodiment further includes:

[0388] The residual of the current image is transformed and quantized to obtain the residual of the current image after transformation and quantization. The transformation can be DCT, DST, DWT or other transformations; and the quantization can be uniform quantization, non-uniform quantization or other quantization.

[0389] In a possible design, the above neural network includes a convolutional network, a uniform pooling layer and a Softmax layer. The convolutional network includes multiple convolutional layers and ReLU layers, where Figure 11A As shown, the probability estimation process of the input data is performed through the neural network to obtain the probability distribution of the residual values ​​of multiple samples of the current image, including:

[0390] S1101. Extract feature information of input data through a convolutional network.

[0391] The input data includes: the residual of the reference image; or

[0392] the residual of the reference image, the prediction of the reference image, and the prediction of the current image; or,

[0393] Residuals of multiple reference images; or,

[0394] Residuals of multiple reference images, predictions of multiple reference images, and predictions of the current image.

[0395] The prediction of the reference image is obtained by predicting the reconstructed sample values ​​of an image coded before the reference image, or by upsampling the reconstructed sample values ​​of a low-resolution image of the reference image to obtain a sampled image, the resolution of which is the same as that of the reference image, and the sampled image is the prediction of the reference image. The low-resolution image of the reference image can be obtained by downsampling the reference image.

[0396] S1102: Probability estimation is performed on the feature information extracted by the convolutional network based on the uniform pooling layer and the Softmax layer, and a probability distribution of residual values ​​of multiple samples included in the residual of the current image is output.

[0397] In a possible design, when encoding the feature map of the residual of the current image, in order to improve the encoding accuracy, super prior information is introduced when calculating the probability distribution of the residual values ​​of multiple samples included in the residual of the current image; the above-mentioned neural network includes a convolutional network, a uniform pooling layer, a first convolutional layer splicing layer and a Softmax layer, and the convolutional network includes multiple convolutional layers and ReLU layers, wherein, as Figure 11B As shown, the probability estimation process of the input data is performed through the neural network to obtain the probability distribution of the residual values ​​of multiple samples of the current image, including:

[0398] S1103. Extract feature information of the residual and super-prior information of the reference image through a convolutional network.

[0399] Among them, the convolutional neural network extracts the feature information of the residual of the reference image and the feature information of the super prior information respectively.

[0400] S1104: Probability estimation is performed on the feature information extracted by the convolutional network based on the uniform pooling layer, the first convolutional layer, the splicing layer, and the Softmax layer, and a probability distribution of residual values ​​of multiple samples included in the residual of the current image is output.

[0401] It should be noted here that the specific process of processing input data through the neural network can be found in the description of the function of the neural network 210 for probability estimation, which will not be described again here.

[0402] In one possible design, the super-prior information is obtained as follows:

[0403] The residual feature map of the current image is input into the super-prior encoding network for feature extraction to obtain the feature vector of the super-prior information; the feature vector of the super-prior information is rounded to obtain the integerized feature vector of the super-prior information; the integerized feature vector of the super-prior information is input into the super-prior decoding network for processing to obtain the super-prior information.

[0404] Among them, the function of the super prior encoding network and the function of the super prior decoding network are opposite.

[0405] In order to enable the decoding side to obtain the super a priori information, after obtaining the integerized feature vector of the super a priori information, arithmetic entropy coding is performed on the integerized feature vector of the super a priori information to obtain a super a priori bitstream.

[0406] S1003 : Perform arithmetic entropy coding on first entropy coded data based on a probability distribution of residual values ​​of a plurality of samples included in the residual of the current image to obtain a code stream of the current image, where the first entropy data represents the residual of the current image.

[0407] As can be seen, in the embodiments of the present application, the residual of the current image is correlated with the residual of the reference image. Therefore, data including at least the residual of the reference image is input into the neural network to obtain a probability distribution of residual values ​​for multiple samples included in the residual of the current image. This probability distribution is then used to encode the first entropy data representing the residual of the current image, which is beneficial for improving encoding accuracy. When determining the probability distribution of the residual values ​​for multiple samples included in the residual of the current image, the introduction of super-prior information can obtain a more accurate probability distribution of the residual values ​​for multiple samples included in the residual of the current image, thereby improving encoding accuracy.

[0408] Figure 12 1 is a flow chart illustrating a process 1200 of a decoding method according to an embodiment of the present application. The process 1200 may be performed by the video decoder 30. The process 1200 is described as a series of steps or operations. It should be understood that the process 1200 may be performed in various orders and / or may occur simultaneously, not limited to Figure 12 The execution order shown.

[0409] like Figure 12 As shown, the decoding method includes:

[0410] S1201: Receive a code stream of a current image.

[0411] S1202. Probability estimation processing is performed on input data through a neural network to obtain a probability distribution of residual values ​​of multiple samples included in the residual of the current image, where the input data at least includes the residual of a reference image, which is a decoded image before decoding the current image.

[0412] The residual of the reference image includes a plurality of samples, and the residual of the reference image includes a plurality of differences between reconstructed values ​​of the plurality of samples and predicted values ​​of the plurality of samples.

[0413] The prediction of the reference image includes prediction values ​​of multiple samples. The residual of the reference image is obtained by predicting the reference image based on the reconstructed sample values ​​of a decoded image that precedes the reference image, or by upsampling the reconstructed sample values ​​of a low-resolution image of the reference image to obtain a sampled image. The resolution of the sampled image is the same as that of the reference image, and the sampled image is the prediction of the reference image. The low-resolution image of the reference image can be obtained by downsampling the reference image.

[0414] Optionally, the first image and the current image may be consecutive or non-consecutive frames; for example, the first image may be the n-1th frame and the current image may be the nth frame; or the first image may be the n-2th frame and the current image may be the nth frame; or, for example, the difference between the timestamp of the current image and the timestamp of the encoded image may be a preset threshold. A low-resolution image of the current image may be obtained by downsampling the current image. The timestamp of the image may be the time at which the image was captured.

[0415] In one possible design, the neural network is a trained neural network, and the method of this embodiment further includes:

[0416] Acquire training input data, where the training input data includes a decoded image before decoding a sample image, where the sample image is a known decoded image, and perform probability estimation processing on the training input data through a neural network to obtain an estimated probability distribution of residual values ​​of multiple samples included in the residual of the sample image; obtain a loss value based on the known probability distribution and the estimated probability distribution of the residual values ​​of multiple samples included in the residual of the sample image; when the loss value converges to a first threshold, or the current number of training times of the neural network is greater than or equal to a second threshold, determine that the current neural network is the neural network corresponding to when the neural network training is completed.

[0417] The input data for training includes: residual values ​​of a plurality of samples included in the residual of the first reference image, or;

[0418] residual values ​​of a plurality of samples included in the residual of the first reference image and residual values ​​of a plurality of samples included in the residual of the second reference image, or;

[0419] prediction values ​​of a plurality of samples included in the prediction of the sample image, residual values ​​of a plurality of samples included in the residual of the first reference image, and prediction values ​​of a plurality of samples included in the prediction of the first reference image; or;

[0420] Prediction values ​​of multiple samples included in the prediction of the sample image, residual values ​​of multiple samples included in the residual of the first reference image, prediction values ​​of multiple samples included in the prediction of the first reference image, prediction values ​​of multiple samples included in the prediction of the second reference image, and residual values ​​of multiple samples included in the residual of the second reference image;

[0421] The first reference image and the second reference image are decoded images before the sample image is decoded. The specific training process can refer to the above method.

[0422] Optionally, the above probability distribution represents the probability distribution of multiple difference values ​​between the original values ​​of multiple samples included in the current image and the predicted values ​​of the multiple samples, or represents the probability distribution of the residual values ​​of each sample point in the multiple samples of the current image, or represents the corresponding probability distribution of the residual values ​​of multiple samples of the current image.

[0423] In a possible design, the above neural network includes a convolutional network, a uniform pooling layer and a Softmax layer. The convolutional network includes multiple convolutional layers and ReLU layers, where Figure 12A As shown, the probability estimation process of the input data is performed through the neural network to obtain the probability distribution of the residual values ​​of multiple samples of the current image, including:

[0424] S1201a. Extract feature information of input data through a convolutional network.

[0425] The input data includes: the residual of the reference image; or

[0426] the residual of the reference image, the prediction of the reference image, and the prediction of the current image; or,

[0427] Residuals of multiple reference images; or,

[0428] Residuals of multiple reference images, predictions of multiple reference images, and predictions of the current image.

[0429] The prediction of the reference image is obtained by predicting the reconstructed sample values ​​of a decoded image before the reference image, or by upsampling the reconstructed sample values ​​of a low-resolution image of the reference image to obtain a sampled image, the resolution of which is the same as that of the reference image, and the sampled image is the prediction of the reference image. The low-resolution image of the reference image can be obtained by downsampling the reference image.

[0430] S1202a: Probability estimation is performed on the feature information extracted by the convolutional network based on the uniform pooling layer and the Softmax layer, and a probability distribution of residual values ​​of multiple samples included in the residual of the current image is output.

[0431] In a possible design, when decoding the feature map of the residual of the current image, in order to improve the decoding accuracy, when calculating the probability distribution of the residual values ​​of multiple samples included in the residual of the current image, super prior information is introduced; the above neural network includes a convolutional network, a uniform pooling layer, a first convolutional layer splicing layer and a Softmax layer, and the convolutional network includes multiple convolutional layers and ReLU layers, wherein, Figure 12B As shown, the probability estimation process of the input data is performed through the neural network to obtain the probability distribution of the residual values ​​of multiple samples of the current image, including:

[0432] S1203a. Extract feature information of the residual and super-prior information of the reference image through a convolutional network.

[0433] Among them, the convolutional neural network extracts the feature information of the residual of the reference image and the feature information of the super prior information respectively.

[0434] S1204a, performing probability estimation on the feature information extracted by the convolutional network based on the uniform pooling layer, the first convolutional layer, the splicing layer, and the Softmax layer, and outputting the probability distribution of the residual values ​​of the multiple samples included in the residual of the current image.

[0435] It should be noted here that the specific process of processing input data through the neural network can be found in the description of the function of the neural network 210 for probability estimation, which will not be described again here.

[0436] In one possible design, the super-prior information is obtained as follows:

[0437] The super-prior bitstream is subjected to arithmetic entropy decoding to obtain an integerized feature vector of the super-prior information. This integerized feature vector of the super-prior information is then processed through a super-prior decoding network to obtain the super-prior information. The functions of the super-prior encoding network and the super-prior decoding network are opposite.

[0438] S1203. Perform arithmetic entropy decoding on the code stream based on a probability distribution of residual values ​​of multiple samples included in the residual of the current image to obtain first entropy decoded data, where the first entropy decoded data represents the residual of the current image; and obtain a reconstructed sample value of the current image based on the residual of the current image.

[0439] It should be pointed out that the first entropy decoded data may be part of the entropy decoded data decoded from the bitstream. After all, the bitstream contains not only residual data but also other data, and other entropy decoded data may be obtained from the bitstream by entropy decoding, so “first” is added as a limitation.

[0440] In one possible design, the first entropy decoded data includes the residual of the current image, or a feature map of the residual of the current image, or the residual of the current image after transformation and quantization.

[0441] Optionally, when the first entropy decoded data is a feature map of the residual of the current image, the method of this embodiment further includes: passing the feature map of the residual of the current image through a decoding network to obtain the residual of the current image.

[0442] It should be noted that the function of the decoding network is opposite to that of the encoding network, so the structure of the decoding network can refer to Figure 6D 、 Figure 6E 、 Figure 6F and Figure 6G The structure of the coding network shown will not be described here.

[0443] Optionally, when the first decoded data is a residual of the current image after transformation and quantization, the method of this embodiment further includes:

[0444] The residual of the transformed and quantized current image is inversely quantized and inversely transformed to obtain the residual of the current image. The inverse transformation may be inverse DCT, inverse DST, inverse DWT, or other inverse transformations; and the inverse quantization may be uniform inverse quantization, non-uniform inverse quantization, or other inverse quantizations.

[0445] It should be noted here that arithmetic entropy decoding and arithmetic entropy coding are a pair of mutually inverse processes. The arithmetic entropy decoding process can refer to the above entropy coding process and will not be described here.

[0446] As can be seen, in the embodiments of the present application, the residual of the current image is correlated with the residual of the reference image. Therefore, data including at least the residual of the reference image is input into the neural network to obtain a probability distribution of residual values ​​for multiple samples included in the residual of the current image. This probability distribution is used to decode the first entropy data representing the residual of the current image, which is beneficial for improving decoding accuracy. When determining the probability distribution of the residual values ​​for multiple samples included in the residual of the current image, introducing super-prior information can obtain a more accurate probability distribution of the residual values ​​for multiple samples included in the residual of the current image, thereby improving decoding accuracy.

[0447] The encoding process and decoding process are comprehensively introduced below with reference to the accompanying drawings.

[0448] See also Figure 13 , Figure 13 This is a schematic diagram of an encoding and decoding process provided in an embodiment of the present application. Figure 13As shown, the encoding device obtains the original residual of the current image, which is a plurality of differences between the original values ​​of multiple samples included in the current image and the predicted values ​​of the multiple samples; the specific process of the original residual of the current image can be referred to the above-mentioned related description and will not be described here; the encoding device uses the encoding network to extract features of the original residual of the current image to obtain a feature map Y of the original residual of the current image; the feature map Y of the residual is processed by the integer module to obtain an integerized feature map Y'; the encoding device uses an arithmetic encoder to perform arithmetic entropy coding on the integerized feature map Y' based on the probability distribution of the residual values ​​of the multiple samples included in the residual of the current image to obtain a code stream of the current image; before this, the encoding device inputs the reconstructed residual of the reference image into a neural network for probability estimation to obtain a probability distribution of the residual values ​​of the multiple samples included in the residual of the current image; it should be noted that the residual of the current image mentioned in this embodiment is also the original residual of the current image mentioned above; the reconstructed residual of the reference image is a plurality of differences between the reconstructed values ​​of the multiple samples included in the reference image and the predicted values ​​of the multiple samples.

[0449] Optionally, the encoding device inputs the reconstructed residual and super-prior information of the reference image into a neural network for probability estimation (i.e., the above-mentioned neural network) to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image; before this, the encoding device uses the super-prior encoding network to perform super-prior feature extraction on the feature map of the residual of the current image to obtain the feature vector Z of the super-prior information; the feature vector Z of the super-prior information is rounded to obtain the integerized feature vector Z' of the super-prior information; the encoding device then uses the super-prior decoding network to process the integerized feature vector of the super-prior information to obtain the super-prior information of the current image.

[0450] In order to facilitate the decoding device to use the super a priori information during decoding, the encoding device uses an arithmetic encoder to encode the integerized feature vector of the super a priori information to obtain a code stream of the super a priori information.

[0451] The decoding device inputs the reconstructed residual of the reference image into a neural network for probability estimation (i.e., the above-mentioned neural network) for processing to obtain a probability distribution of residual values ​​of multiple samples included in the reconstructed residual of the current image; after obtaining the code stream of the current image, the decoding device uses an arithmetic decoder to perform arithmetic entropy decoding on the code stream of the current image based on the probability distribution of residual values ​​of multiple samples included in the reconstructed residual of the current image to obtain a feature map of the reconstructed residual of the current image; and uses the decoding network to process the feature map of the reconstructed residual of the current image to obtain the reconstructed residual of the current image.

[0452] Optionally, the decoding device uses an arithmetic decoder to decode the code stream of the super-prior information to obtain a feature vector of the super-prior information, and uses a super-prior decoding network to process the feature vector of the super-prior information to obtain the super-prior information; the super-prior information and the reconstructed residual of the reference frame image are input into a neural network for probability estimation for processing to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image.

[0453] See also Figure 14 , Figure 14 This is a schematic diagram of another encoding and decoding process provided in an embodiment of the present application. Figure 14 As shown, the encoding device obtains the original residual of the current image, which is a plurality of differences between the original values ​​of a plurality of samples included in the current image and the predicted values ​​of the plurality of samples; the specific process of the original residual of the current image can be referred to the above-mentioned related description, which will not be described here; the encoding device uses the encoding network to extract features of the residual of the current image to obtain a feature map Y of the residual ResiPicN of the current image; the feature map of the residual of the current image is processed by the integer module to obtain an integerized feature map Y'; the encoding device uses an arithmetic encoder to perform feature extraction on the integerized feature map Y' based on the probability distribution of the residual values ​​of the plurality of samples included in the residual of the current image Arithmetic entropy coding is performed to obtain a code stream of the current image; before this, the encoding device uses a recurrent neural network to obtain hidden information Hn based on the reconstructed residuals of multiple reference images. The specific process can be found in the relevant description of the above embodiment and will not be described here; the encoding device also inputs the hidden information Hn into a neural network for probability estimation (i.e., the above neural network) to obtain a probability distribution of residual values ​​of multiple samples included in the residual of the current image; it should be pointed out that the residual of the current image mentioned in this embodiment is also the original residual of the above current image; the reconstructed residual of the reference image is a plurality of differences between the reconstructed values ​​of multiple samples included in the reference image and the predicted values ​​of the multiple samples.

[0454] Optionally, the encoding device inputs the hidden information Hn and the super-prior information into a neural network for probability estimation to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image; before this, the encoding device uses the super-prior encoding network to perform super-prior feature extraction on the feature map of the residual of the current image to obtain the feature vector Z of the super-prior information; the feature vector Z of the super-prior information is rounded to obtain the integerized feature vector Z' of the super-prior information; the encoding device then uses the super-prior decoding network to process the integerized feature vector of the super-prior information to obtain the super-prior information of the current image.

[0455] In order to facilitate the decoding device to use the super a priori information during decoding, the encoding device uses an arithmetic encoder to encode the integerized feature vector of the super a priori information to obtain a code stream of the super a priori information.

[0456] The decoding device uses a recurrent neural network to obtain hidden information Hn based on the reconstructed residuals of multiple reference images. The specific process can be found in the relevant description of the above embodiment and will not be described here again; the hidden information Hn is input into the neural network for probability estimation for processing to obtain the probability distribution of the residual values ​​of multiple samples included in the reconstructed residual of the current image; after obtaining the code stream of the current image, the decoding device uses an arithmetic decoder to perform arithmetic entropy decoding on the code stream of the current image based on the probability distribution of the residual values ​​of multiple samples included in the reconstructed residual of the current image to obtain a feature map of the reconstructed residual of the current image; the feature map of the reconstructed residual of the current image is processed using the decoding network to obtain the reconstructed residual of the current image.

[0457] Optionally, the decoding device uses an arithmetic decoder to decode the code stream of the super-prior information to obtain a feature vector of the integerized super-prior information, and uses a super-prior decoding network to process the feature vector of the integerized super-prior information to obtain super-prior information; the super-prior information and the hidden information Hn are input into a neural network for probability estimation for processing to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image.

[0458] See also Figure 15 , Figure 15 This is a schematic diagram of another encoding and decoding process provided in an embodiment of the present application. Figure 15 As shown, the encoding device obtains the original residual of the current image, which is a plurality of differences between the original values ​​of the multiple samples included in the current image and the predicted values ​​of the multiple samples. The specific process of the original residual of the current image can be referred to the above-mentioned related description and will not be described here again; the encoding device uses the encoding network to extract features of the original residual of the current image to obtain a feature map Y of the original residual of the current image; the feature map Y of the original residual of the current image is processed by the integer module to obtain an integer feature map Y'; the encoding device extracts the features of the multiple samples included in the residual of the current image based on the residual of the current image. The probability distribution of the value is obtained by using an arithmetic encoder to perform arithmetic entropy coding on the integerized feature map Y' to obtain a code stream of the current image; before this, the encoding device inputs the reconstructed residual of the reference image, the prediction of the reference image and the prediction of the current image into a neural network for probability estimation to obtain a probability distribution of the residual values ​​of multiple samples included in the residual of the current image; it should be pointed out that the residual of the current image mentioned in this embodiment is also the original residual of the current image mentioned above; the reconstructed residual of the reference image is a plurality of differences between the reconstructed values ​​of multiple samples included in the reference image and the predicted values ​​of the multiple samples.

[0459] Optionally, the encoding device inputs the reconstructed residual of the reference image, the prediction of the reference image, the prediction of the current image and the super-prior information into a neural network for probability estimation to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image; before this, the encoding device uses the super-prior encoding network to perform super-prior feature extraction on the feature map of the residual to obtain the feature vector Z of the super-prior information; the feature vector Z of the super-prior information is rounded to obtain the integerized feature vector Z' of the super-prior information; the encoding device then uses the super-prior decoding network to process the integerized feature vector of the super-prior information to obtain the super-prior information of the current image.

[0460] In order to facilitate the decoding device to use the super a priori information during decoding, the encoding device uses an arithmetic encoder to encode the integerized feature vector of the super a priori information to obtain a code stream of the super a priori information.

[0461] The decoding device inputs the reconstructed residual of the reference image, the prediction of the reference image, and the prediction of the current image into a neural network for probability estimation for processing, thereby obtaining a probability distribution of residual values ​​of multiple samples included in the reconstructed residual of the current image; after obtaining the code stream of the current image, the decoding device uses an arithmetic decoder to perform arithmetic entropy decoding on the code stream of the current image based on the probability distribution of residual values ​​of multiple samples included in the reconstructed residual of the current image, thereby obtaining a feature map of the reconstructed residual of the current image; and uses the decoding network to process the feature map of the reconstructed residual of the current image, thereby obtaining the reconstructed residual of the current image.

[0462] Optionally, the decoding device uses an arithmetic decoder to decode the code stream of the super-prior information to obtain a feature vector of the integerized super-prior information, and uses a super-prior decoding network to process the feature vector of the integerized super-prior information to obtain the super-prior information; the super-prior information, the reconstructed residual of the reference image, the prediction of the reference image, and the prediction of the current image are input into a neural network for probability estimation for processing to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image.

[0463] See also Figure 16 , Figure 16 This is a schematic diagram of another encoding and decoding process provided in an embodiment of the present application. Figure 16As shown, the encoding device obtains the original residual of the current image, which is a plurality of differences between the original values ​​of the multiple samples included in the current image and the predicted values ​​of the multiple samples. The specific process of the original residual of the current image can be found in the above-mentioned related description and will not be described here; the encoding device uses the encoding network to extract features of the original residual of the current image to obtain a feature map Y of the original residual of the current image; the feature map Y of the original residual of the current image is processed by the integer module to obtain an integerized feature map Y'; the encoding device uses an arithmetic encoder to perform arithmetic entropy on the integerized feature map Y' based on the probability distribution of the residual values ​​of the multiple samples included in the residual of the current image Encoding to obtain the code stream of the current image; before this, the encoding device uses a recurrent neural network to obtain hidden information Hn based on the reconstructed residuals of multiple reference images and the predictions of multiple reference images. The specific process can be found in the relevant description of the above embodiment and will not be described here; the encoding device also inputs the hidden information Hn and the prediction of the current image into the neural network for probability estimation to obtain the probability distribution of the residual values ​​of the multiple samples included in the residual of the current image; it should be pointed out that the residual of the current image mentioned in this embodiment is the original residual of the current image mentioned above; the reconstructed residual of the reference image is the multiple differences between the reconstructed values ​​of the multiple samples included in the reference image and the predicted values ​​of the multiple samples.

[0464] Optionally, the encoding device inputs the hidden information Hn, the prediction of the current image and the super-prior information into a neural network for probability estimation to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image; before this, the encoding device uses the super-prior encoding network to perform super-prior feature extraction on the feature map of the residual of the current image to obtain the feature vector Z of the super-prior information; the feature vector Z of the super-prior information is rounded to obtain the integerized feature vector Z' of the super-prior information; the encoding device then uses the super-prior decoding network to process the integerized feature vector of the super-prior information to obtain the super-prior information of the current image.

[0465] In order to facilitate the decoding device to use the super a priori information during decoding, the encoding device uses an arithmetic encoder to encode the integerized feature vector of the super a priori information to obtain a code stream of the super a priori information.

[0466] The decoding device uses a recurrent neural network to obtain hidden information Hn based on the reconstructed residuals of multiple reference images and the predictions of multiple reference images. The specific process can be found in the relevant description of the above embodiment and will not be described here again; the hidden information Hn and the prediction of the current image are input into the neural network for probability estimation for processing to obtain the probability distribution of the residual values ​​of the multiple samples included in the reconstructed residual of the current image; after obtaining the code stream of the current image, the decoding device uses an arithmetic decoder to perform arithmetic entropy decoding on the code stream of the current image based on the probability distribution of the residual values ​​of the multiple samples included in the reconstructed residual of the current image to obtain a feature map of the reconstructed residual of the current image; the feature map of the reconstructed residual of the current image is processed using the decoding network to obtain the reconstructed residual of the current image.

[0467] Optionally, the decoding device uses an arithmetic decoder to decode the code stream of the super-prior information to obtain a feature vector of the integerized super-prior information, and uses a super-prior decoding network to process the feature vector of the integerized super-prior information to obtain the super-prior information; the super-prior information, hidden information Hn and the prediction of the current image are input into a neural network for probability estimation for processing to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image.

[0468] See also Figure 17 , Figure 17 This is a schematic diagram of an encoding and decoding process provided in an embodiment of the present application. Figure 17 As shown, the encoding device obtains the original residual of the current image, which is a plurality of differences between the original values ​​of multiple samples included in the current image and the predicted values ​​of the multiple samples; the specific process of the original residual of the current image can be found in the above-mentioned related description and will not be described here; the encoding device quantizes the original residual of the current image to obtain the quantized original residual of the current image; the encoding device uses an arithmetic encoder to perform arithmetic entropy encoding on the quantized residual of the current image based on the probability distribution of the residual values ​​of multiple samples included in the residual of the current image to obtain the code stream of the current image; it should be pointed out that the residual of the current image mentioned in this embodiment is also the original residual of the current image mentioned above.

[0469] Among them, the input data of the neural network used for probability estimation includes the following cases:

[0470] like Figure 13 As shown, the encoding device inputs the reconstructed residual of the reference image into a neural network for probability estimation to obtain a probability distribution of residual values ​​of multiple samples included in the residual of the current image;

[0471] Or as Figure 14As shown, the encoding device uses a recurrent neural network to obtain hidden information Hn based on the reconstructed residuals of multiple reference images. The specific process can be found in the relevant description of the above embodiment and will not be described here. The encoding device also inputs the hidden information Hn into the neural network for probability estimation to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image.

[0472] Or as Figure 15 As shown, the encoding device inputs the reconstructed residual of the reference image, the prediction of the reference image and the prediction of the current image into a neural network for probability estimation, and obtains the probability distribution of the residual values ​​of multiple samples included in the residual of the current image.

[0473] Or as Figure 16 As shown, the encoding device uses a recurrent neural network to obtain hidden information Hn based on the reconstructed residuals of multiple reference images and the predictions of multiple reference images. The specific process can be found in the relevant description of the above embodiment and will not be described here. The encoding device also inputs the hidden information Hn and the prediction of the current image into the neural network for probability estimation to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image.

[0474] After obtaining the code stream of the current image, the decoding device uses an arithmetic decoder to perform arithmetic entropy decoding on the code stream of the current image based on the probability distribution of the residual value of the current image to obtain the quantized residual of the current image; and dequantizes the quantized residual of the current image to obtain the reconstructed residual of the current image; wherein, Figure 13 As shown, the decoding device inputs the reference reconstruction residual into the neural network for probability estimation for processing to obtain the probability distribution of the residual value of the current image; or, as shown Figure 14 As shown, the decoding device uses a recurrent neural network to obtain hidden information Hn based on the reconstructed residuals of multiple reference images. The specific process can be found in the relevant description of the above embodiment and will not be described here again. The hidden information Hn is input into the neural network for probability estimation for processing to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image; or, as shown in FIG. Figure 15 As shown, the decoding device inputs the reconstructed residual of the reference image, the prediction of the reference image and the prediction of the current image into a neural network for probability estimation for processing, and obtains the probability distribution of the residual values ​​of multiple samples included in the residual of the current image; or, as shown Figure 16 As shown, the decoding device uses a recurrent neural network to obtain hidden information Hn based on the reconstructed residuals of multiple reference images and the predictions of multiple reference images. The specific process can be found in the relevant description of the above embodiment and will not be described here. The hidden information Hn and the prediction input of the current image are processed in a neural network for probability estimation to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image.

[0475] See also Figure 18 , Figure 18 This is a schematic diagram of an encoding and decoding process provided in an embodiment of the present application. Figure 18 As shown, the encoding device obtains the original residual of the current image, which is a plurality of differences between the original values ​​of multiple samples included in the current image and the predicted values ​​of the multiple samples; the specific process of the original residual of the current image can be referred to the above related description and will not be described here; the encoding device transforms (for example, DCT, or DST, DWT) and quantizes the original residual of the current image to obtain the transformed and quantized original residual of the current image; the encoding device performs arithmetic entropy coding on the transformed and quantized original residual of the current image based on the probability distribution of the residual values ​​of the multiple samples included in the residual of the current image using an arithmetic encoder to obtain a code stream of the current image; it should be noted that the residual of the current image mentioned in this embodiment is the original residual of the current image mentioned above;

[0476] Among them, the input data of the neural network used for probability estimation includes the following cases:

[0477] like Figure 13 As shown, the encoding device inputs the reconstructed residual of the reference image into a neural network for probability estimation to obtain a probability distribution of residual values ​​of multiple samples included in the residual of the current image;

[0478] Or as Figure 14 As shown, the encoding device uses a recurrent neural network to obtain hidden information Hn based on the reconstructed residuals of multiple reference images. The specific process can be found in the relevant description of the above embodiment and will not be described here. The encoding device also inputs the hidden information Hn into the neural network for probability estimation to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image.

[0479] Or as Figure 15 As shown, the encoding device inputs the reconstructed residual of the reference image, the prediction of the reference image and the prediction of the current image into a neural network for probability estimation, and obtains the probability distribution of the residual values ​​of multiple samples included in the residual of the current image.

[0480] Or as Figure 16 As shown, the encoding device uses a recurrent neural network to obtain hidden information Hn based on the reconstructed residuals of multiple reference images and the predictions of multiple reference images. The specific process can be found in the relevant description of the above embodiment and will not be described here. The encoding device also inputs the hidden information Hn and the prediction of the current image into the neural network for probability estimation to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image.

[0481] After obtaining the code stream of the current image, the decoding device uses an arithmetic decoder to perform arithmetic entropy decoding on the code stream of the current image based on the probability distribution of the residual value of the current image to obtain the residual of the current image after transformation and quantization; dequantizes and inversely transforms the residual of the current image after transformation and quantization to obtain the reconstructed residual of the current image; wherein, if Figure 13 As shown, the decoding device inputs the reference reconstruction residual into the neural network for probability estimation for processing to obtain the probability distribution of the residual value of the current image; or, as shown Figure 14 As shown, the decoding device uses a recurrent neural network to obtain hidden information Hn based on the reconstructed residuals of multiple reference images. The specific process can be found in the relevant description of the above embodiment and will not be described here again. The hidden information Hn is input into the neural network for probability estimation for processing to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image; or, as shown in FIG. Figure 15 As shown, the decoding device inputs the reconstructed residual of the reference image, the prediction of the reference image and the prediction of the current image into a neural network for probability estimation for processing, and obtains the probability distribution of the residual values ​​of multiple samples included in the residual of the current image; or, as shown Figure 16 As shown, the decoding device uses a recurrent neural network to obtain hidden information Hn based on the reconstructed residuals of multiple reference images and the predictions of multiple reference images. The specific process can be found in the relevant description of the above embodiment and will not be described here. The hidden information Hn and the prediction input of the current image are processed in a neural network for probability estimation to obtain the probability distribution of the residual values ​​of multiple samples included in the residual of the current image.

[0482] It should be noted here that the above encoding network and decoding network can be implemented through a neural network, such as a convolutional neural network.

[0483] Those skilled in the art will appreciate that the functions described in conjunction with the various illustrative logic blocks, modules, and algorithm steps disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions described in the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media can include computer-readable storage media, which corresponds to tangible media, such as data storage media, or communication media including any media that facilitates the transfer of computer programs from one place to another (e.g., based on a communication protocol). In this manner, computer-readable media can generally correspond to (1) non-transitory tangible computer-readable storage media, or (2) communication media, such as signals or carrier waves. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in this application. A computer program product can include computer-readable media.

[0484] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are actually directed to non-transitory tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), and Blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0485] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor," as used herein, may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described by the various illustrative logical blocks, modules, and steps described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements.

[0486] The techniques of this application can be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., a chipset). Various components, modules, or units are described herein to emphasize functional aspects of devices for performing the disclosed techniques, but they do not necessarily require implementation by different hardware units. In fact, as described above, the various units may be combined in a codec hardware unit in conjunction with appropriate software and / or firmware, or provided by interoperating hardware units (including one or more processors as described above).

[0487] The above description is merely an exemplary embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for decoding a video image, characterized in that: include: Receive the code stream of the current image; performing probability estimation processing on input data using a trained neural network to obtain a probability distribution of residual values ​​of a plurality of samples included in the residual of the current image, wherein the input data at least includes a residual of a reference image, the reference image being a decoded image before decoding the current image; Performing arithmetic entropy decoding on the code stream based on a probability distribution of residual values ​​of a plurality of samples included in the residual of the current image to obtain first entropy decoded data, where the first entropy decoded data represents the residual of the current image; Based on the residual of the current image, a reconstructed sample value of the current image is obtained.

2. The method according to claim 1, characterized in that The neural network is a neural network trained through the following process: The method further comprises: Acquire the training input data, wherein the training input data includes a decoded image before decoding a sample image, wherein the sample image is a known decoded image, Performing probability estimation processing on the training input data through a neural network to obtain an estimated probability distribution of residual values ​​of a plurality of samples included in the residual of the sample image; Obtaining a loss value based on a known probability distribution of residual values ​​of a plurality of samples included in the residual of the sample image and the estimated probability distribution; When the loss value converges to a first threshold, or the current training times of the neural network are greater than or equal to a second threshold, the current neural network is determined to be the neural network corresponding to when the neural network training is completed.

3. The method according to claim 2, characterized in that The input data for the training includes: The residual of the first reference image includes residual values ​​of a plurality of samples, or; The residual values ​​of the plurality of samples included in the residual of the first reference image and the residual values ​​of the plurality of samples included in the residual of the second reference image, or; The prediction values ​​of the plurality of samples included in the prediction of the sample image, the residual values ​​of the plurality of samples included in the residual of the first reference image, and the prediction values ​​of the plurality of samples included in the prediction of the first reference image; or; The prediction values ​​of the plurality of samples included in the prediction of the sample image, the residual values ​​of the plurality of samples included in the residual of the first reference image, the prediction values ​​of the plurality of samples included in the prediction of the first reference image, the prediction values ​​of the plurality of samples included in the prediction of the second reference image, and the residual values ​​of the plurality of samples included in the residual of the second reference image; The first reference image and the second reference image are decoded images before decoding the sample image.

4. The method according to any one of claims 1 to 3, characterized in that The probability distribution represents a probability distribution of a plurality of difference values ​​between the reconstructed values ​​of a plurality of samples included in the residual of the current image and the predicted values ​​of the plurality of samples; or, The probability distribution represents the probability distribution of the residual value of each sample in the multiple samples of the current image.

5. The method according to any one of claims 1 to 3, characterized in that The probability distribution represents the probability distribution of residual values ​​of multiple samples of the current image.

6. The method according to claim 1, wherein The first entropy decoded data is the residual of the current image, or a feature map of the residual of the current image.

7. The method according to claim 1, characterized in that The first entropy decoded data is the residual of the current image after transformation and quantization.

8. The method according to claim 6, characterized in that The first entropy decoded data is a feature map of the residual of the current image, and the method further includes: The residual feature map of the current image is passed through a decoding network to obtain the residual of the current image.

9. The method according to claim 7, characterized in that The first entropy decoded data is the residual of the current image after transformation and quantization, and the method further includes: Performing inverse transformation and inverse quantization on the transformed and quantized residual of the current image to obtain the residual of the current image.

10. The method according to claim 8, characterized in that The input data further includes hyper-prior information, The neural network includes a convolutional network, a uniform pooling layer, a first convolutional layer, a concatenation layer, and a normalized exponential function Softmax layer; the convolutional network includes multiple convolutional layers and an excitation ReLU layer, wherein: The performing probability estimation processing on the input data by the neural network to obtain the probability distribution of the residual of the current image includes: The convolutional network is used to extract the residual of the reference image and the feature information in the super-prior information respectively; the feature information extracted by the convolutional network is probability estimated through the concat layer, the first convolutional layer, the uniform pooling layer and the Softmax layer, and the probability distribution of the residual values ​​of multiple samples included in the residual of the current image is output.

11. The method according to claim 1, characterized in that The input data also includes a prediction of the reference image and a prediction of the current image; The neural network includes a convolutional network, a uniform pooling layer, and a normalized exponential function Softmax layer; the convolutional network includes multiple convolutional layers and an excitation ReLU layer, wherein: The performing probability estimation processing on the input data by the neural network to obtain the probability distribution of residual values ​​of the plurality of samples included in the residual of the current image includes: The convolutional network is used to extract the residual of the reference image, the prediction of the reference image, and the predicted feature information of the current image; the uniform pooling layer and the Softmax layer are used to perform probability estimation on the feature information extracted by the convolutional network, and the probability distribution of the residual values ​​of multiple samples included in the residual of the current image is output.

12. The method according to claim 1, characterized in that The input data includes: residuals of a plurality of reference images, where the plurality of reference images are a plurality of decoded images before decoding the current image; The neural network includes a convolutional network, a uniform pooling layer, and a normalized exponential function Softmax layer; the convolutional network includes multiple convolutional layers and an excitation ReLU layer, wherein: The performing probability estimation processing on the input data by the neural network to obtain the probability distribution of residual values ​​of the plurality of samples included in the residual of the current image includes: The convolutional network is used to extract feature information of the residuals of the multiple reference images; the feature information extracted by the convolutional network is probability estimated through the uniform pooling layer and the softmax layer, and the probability distribution of the residual values ​​of the multiple samples included in the residual of the current image is output.

13. The method according to claim 1, wherein The input data includes: residuals of a plurality of reference pictures, predictions of the plurality of reference pictures, and a prediction of a current picture, wherein the plurality of reference pictures are a plurality of decoded pictures before decoding the current picture; The neural network includes a convolutional network, a uniform pooling layer, and a normalized exponential function Softmax layer; the convolutional network includes multiple convolutional layers and an excitation ReLU layer, wherein: The performing probability estimation processing on the input data by the neural network to obtain the probability distribution of residual values ​​of the plurality of samples included in the residual of the current image includes: The convolutional network is used to extract the residuals of the multiple reference images and the predictions of the multiple reference images and the predicted feature information of the current image; the feature information extracted by the convolutional network is probability estimated through the uniform pooling layer and the Softmax layer, and the probability distribution of the residual values ​​of the multiple samples included in the residual of the current image is output.

14. A method for encoding a video image, characterized in that: include: Obtaining a residual of a current image, the current image including a plurality of samples, the residual being a difference between an original value of the sample and a predicted value of the sample; performing probability estimation processing on input data using a trained neural network to obtain a probability distribution of residual values ​​of a plurality of samples included in the residual of the current image, wherein the input data at least includes a residual of a reference image, the reference image being an encoded image before encoding the current image; Performing arithmetic entropy coding on the first entropy coded data according to a probability distribution of residual values ​​of a plurality of samples included in the residual of the current image to obtain a code stream of the current image, wherein the first entropy coded data represents the residual of the current image.

15. The method according to claim 14, characterized in that The neural network is trained through the following process: The method further comprises: Acquire the training input data, wherein the training input data includes an encoded image before encoding a sample image, and the sample image is a known encoded image; Performing probability estimation processing on the training input data through a neural network to obtain an estimated probability distribution of residual values ​​of a plurality of samples included in the residual of the sample image; Obtaining a loss value based on a known probability distribution of residual values ​​of a plurality of samples included in the residual of the sample image and the estimated probability distribution; When the loss value converges to a first threshold, or the current training times of the neural network are greater than or equal to a second threshold, the current neural network is determined to be the neural network corresponding to when the neural network training is completed.

16. The method according to claim 15, characterized in that The input data for the training includes: The residual of the first reference image includes residual values ​​of a plurality of samples, or; The residual values ​​of the plurality of samples included in the residual of the first reference image and the residual values ​​of the plurality of samples included in the residual of the second reference image, or; The prediction values ​​of the plurality of samples included in the prediction of the sample image, the residual values ​​of the plurality of samples included in the residual of the first reference image, and the prediction values ​​of the plurality of samples included in the prediction of the first reference image; or; The prediction values ​​of the plurality of samples included in the prediction of the sample image, the residual values ​​of the plurality of samples included in the residual of the first reference image, the prediction values ​​of the plurality of samples included in the prediction of the first reference image, the prediction values ​​of the plurality of samples included in the prediction of the second reference image, and the residual values ​​of the plurality of samples included in the residual of the second reference image; The first reference image and the second reference image are encoded images before encoding the sample image.

17. The method according to any one of claims 14 to 16, characterized in that: The probability distribution represents a probability distribution of a plurality of difference values ​​between original values ​​of a plurality of samples included in the residual of the current image and predicted values ​​of the plurality of samples; or, The probability distribution represents the probability distribution of the residual value of each sample in the multiple samples of the current image.

18. The method according to any one of claims 14 to 16, characterized in that: The probability distribution represents the probability distribution corresponding to the residual values ​​of multiple samples of the current image.

19. The method according to claim 14, wherein The first entropy coded data includes a residual of the current image, or a feature map of the residual of the current image.

20. The method according to claim 14, wherein The first entropy coded data includes a transformed and quantized residual of the current image.

21. The method according to claim 19, wherein The first entropy coded data is a feature map of the residual of the current image, and the method further includes: The residual of the current image is passed through an encoding network to obtain a feature map of the residual of the current image.

22. The method according to claim 20, characterized in that The first entropy coded data is the residual of the current image after transformation and quantization, and the method further includes: The residual of the current image is transformed and quantized to obtain the transformed and quantized residual of the current image.

23. The method according to claim 21, characterized in that The input data further includes hyper-prior information, The neural network includes a convolutional network, a uniform pooling layer, a first convolutional layer, a concatenation layer, and a normalized exponential function Softmax layer; the convolutional network includes multiple convolutional layers and an excitation ReLU layer, wherein: The performing probability estimation processing on the input data by the neural network to obtain the probability distribution of residual values ​​of the plurality of samples included in the residual of the current image includes: The convolutional network is used to extract the residual of the reference image and the feature information in the super-prior information; the feature information extracted by the convolutional network is probability estimated through the uniform pooling layer, the first convolutional layer, the concat layer and the Softmax layer, and the probability distribution of the residual values ​​of multiple samples included in the residual of the current image is output.

24. The method according to claim 23, wherein The method further comprises: Inputting the residual feature map of the current image into a super-prior coding network to obtain a feature vector of the super-prior information; Rounding the eigenvector of the super a priori information to obtain an integerized eigenvector of the super a priori information; The integerized feature vector of the super-prior information is input into a super-prior decoding network to obtain the super-prior information.

25. The method according to claim 14, wherein The input data also includes a prediction of the reference image and a prediction of the current image; The neural network includes a convolutional network, a uniform pooling layer, and a normalized exponential function Softmax layer; the convolutional network includes multiple convolutional layers and an excitation ReLU layer, wherein: The performing probability estimation processing on the input data by the neural network to obtain the probability distribution of residual values ​​of the plurality of samples included in the residual of the current image includes: The convolutional network is used to extract the residual of the reference image, the prediction of the reference image, and the predicted feature information of the current image; the uniform pooling layer and the softmax layer are used to perform probability estimation on the feature information extracted by the convolutional network, and the probability distribution of the residual values ​​of multiple samples of the current image is output.

26. The method according to claim 14, wherein The input data includes residuals of multiple reference images, prediction signals of multiple reference images, and a prediction signal of the current image; the multiple reference images are multiple encoded images before encoding the current image, The neural network includes a convolutional network, a uniform pooling layer, and a normalized exponential function Softmax layer; the convolutional network includes multiple convolutional layers and an excitation ReLU layer, wherein: The performing probability estimation processing on the input data by the neural network to obtain the probability distribution of residual values ​​of the plurality of samples included in the residual of the current image includes: The convolutional network is used to extract the residuals of the multiple reference images, the predictions of the multiple reference images, and the predicted feature information of the current image; the uniform pooling layer and the Softmax layer are used to perform probability estimation on the feature information extracted by the convolutional network, and the probability distribution of the residual values ​​of the multiple samples included in the residual of the current image is output.

27. The method according to claim 14, wherein The input data includes residuals of a plurality of reference images, where the plurality of reference images are a plurality of encoded images before encoding the current image. The neural network includes a convolutional network, a uniform pooling layer, and a normalized exponential function Softmax layer; the convolutional network includes multiple convolutional layers and an excitation ReLU layer, wherein: The performing probability estimation processing on the input data by the neural network to obtain the probability distribution of residual values ​​of the plurality of samples included in the residual of the current image includes: The convolutional network is used to extract feature information of the residuals of the multiple reference images; the feature information extracted by the convolutional network is probability estimated through the uniform pooling layer and the softmax layer, and the probability distribution of the residual values ​​of the multiple samples included in the residual of the current image is output.

28. A decoder, characterized in that The method comprises a processing circuit for executing the method according to any one of claims 1 to 13.

29. An encoder, characterized in that comprising a processing circuit for executing the method according to any one of claims 14-27.

30. A computer program product, characterized in that The method comprises a program code, which, when executed on a computer or a processor, is used to perform the method according to any one of claims 1 to 27.

31. A decoder, characterized in that include: one or more processors; A non-transitory computer-readable storage medium is coupled to the processor and stores a program executed by the processor, wherein when the program is executed by the processor, the decoder performs the method according to any one of claims 1 to 13.

32. An encoder, characterized in that include: one or more processors; A non-transitory computer-readable storage medium is coupled to the processor and stores a program executed by the processor, wherein when the program is executed by the processor, the encoder performs the method according to any one of claims 14 to 27.

33. A non-transitory computer-readable storage medium, characterized in that The invention comprises program codes, which, when executed by a computer device, are used to perform the method according to any one of claims 1 to 27.

Images